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ABSTRACT 


The  Navy’s  Strategic  Planning  and  Analysis  Directorate  (OPNAV  N14)  uses  a 
complex  model  to  project  officer  status  in  the  coming  years.  The  Officer  Strategic 
Analysis  Model  (OSAM)  projects  officer  status  using  an  initial  inventory,  historical 
loss  rates,  and  dependent  functions  for  accessions,  losses,  lateral  transfers,  and 
promotions  that  reflect  Navy  policy  and  U.S.  law.  OSAM  is  a  tool  for  informing 
decision  makers  as  they  consider  potential  policy  changes,  or  analyze  the  impact 
of  policy  changes  already  in  place,  by  generating  Navy  Officer  inventory 
projections  for  a  specified  time  horizon. 

This  research  explores  applications  of  data  farming  for  potential 
improvement  of  OSAM.  An  analysis  of  OSAM  inventory  forecast  variations  over  a 
large  number  of  scenarios  while  changing  multiple  input  parameters  enables 
assessment  of  key  inputs.  This  research  explores  OSAM  through  applying  the 
principles  of  design  of  experiments,  regression  modeling,  and  nonlinear 
programming.  The  objectives  of  this  portion  of  the  work  include  identifying  critical 
parameters,  determining  a  suitable  measure  of  effectiveness,  assessing  model 
sensitivities,  evaluating  performance  across  a  spectrum  of  loss  adjustment 
factors,  and  determining  appropriate  values  of  key  model  inputs  for  future  use  in 
forecasting  Navy  officer  inventory. 
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EXECUTIVE  SUMMARY 


Manpower  and  personnel  costs  consume  a  significant  portion  of  the  U.S.  Navy 
budget  every  year,  and  Navy  planners  face  the  challenge  of  balancing  manpower 
requirements  and  mandated  end  strength  with  budget  constraints.  The  variability 
of  human  behavior  further  complicates  the  issue  of  forecasting  strength.  The 
Chief  of  Naval  Personnel’s  (NTs)  Strategic  Resourcing  Branch  is  responsible  for 
analyzing  manpower  inventory  forecasts  and  estimating  the  Navy’s  manpower 
expenditures  to  be  included  in  the  budget  and  Program  Objectives  Memorandum 
(POM)  submitted  to  the  Secretary  of  the  Navy  every  two  years.  Forecasting 
Navy  Officer  inventory  is  a  complex  problem  with  an  imperfect  solution.  One  tool 
currently  in  use  to  tackle  this  problem  is  the  Officer  Strategic  Analysis  Model 
(OSAM),  a  model  that  follows  individual  officers  from  an  initial  inventory,  or 
entities,  through  possible  attribute  changes  during  a  forecast  period,  resulting  in 
a  projection  of  status  for  each  officer  in  the  coming  years.  OSAM  is  a  tool  for 
informing  decision  makers  as  they  consider  potential  policy  changes,  or 
analyzing  the  impact  of  policy  changes  already  in  place,  by  generating  Navy 
officer  inventory  projections  for  a  specified  time  horizon. 

Navy  officer  inventory  changes  continuously,  but  decision  makers  find  it 
useful  to  have  accurate  information  about  annual  variations  in  inventory;  this  is 
what  OSAM  models.  The  key  attributes  of  rank,  designator,  time-in-service,  and 
time-in-grade  describe  each  officer  in  inventory  and  adjust  as  time  elapses.  In 
addition  to  initial  inventory,  functions  describing  accession,  promotion,  lateral 
transfer,  and  loss  influence  the  total  officer  inventory  characterization.  To  model 
losses,  OSAM  multiplies  historical  loss  rates  by  a  loss  adjustment  factor  for  each 
category  of  officers.  The  model  is  capable  of  assigning  a  unique  loss  adjustment 
factor  for  each  combination  of  key  attributes,  but  recent  practice  is  to  apply  the 
same  loss  adjustment  factor  universally  to  all  designators  and  grades  in  a  given 
projection  year  (i.e.,  loss  adjustment  factor  =  1.041  in  year  two,  1.059  in  year 
three,  1.085  in  year  four,  1.114  in  year  five).  These  values  describe  a  set  of 


xv 


typical  loss  adjustment  factors  that  OPNAV  N14  provided  to  model  a  slowly 
improving  economy.  The  thesis  work  presented  here  analyzes  OSAM  using 
designs  of  experiments  and  simulation  analysis  to  explore  the  capabilities  and 
limitations  of  the  model,  specifically  targeting  the  lateral  transfer  and  loss 
functions  within  the  model. 

Deployment  of  OSAM  on  Naval  Postgraduate  School  (NPS)  computers 
enables  exploitation  of  multiple  processors  and  advanced  statistical  methods  of 
analysis.  Several  different  analytical  methods  provide  insights  into  OSAM.  Final 
analysis  results  in  recommended  loss  adjustment  factors  that  generate  a  better 
forecast  than  past  practice,  at  a  90%  confidence  level.  An  OSAM  modification 
allows  the  tracking  of  loss  behavior  of  Surface  Warfare  Officers  (SWOs)  whose 
lateral  transfer  efforts  are  unsuccessful.  Analysis  of  experiments  reveals  that 
these  officers’  loss  adjustment  factors  are  the  same  as  SWOs  who  never  applied 
for  lateral  transfer,  at  a  95%  confidence  level.  Additional  experiments  are 
necessary  to  determine  whether  there  is  a  benefit  to  modeling  these  officers 
separately,  as  this  OSAM  modification  allows  a  unique  historical  loss  rate  to 
enter  the  model. 

This  thesis  compares  OSAM  forecasts  to  historical  inventories  over  a  five 
year  period,  and  results  could  potentially  improve  with  additional  base  year  data 
available.  Extension  of  this  analysis  over  a  broader  time  frame  may  also  capture 
the  impact  of  varying  political  and  economic  environments.  This  first  foray  into 
data  farming  OSAM  studies  only  two  officer  communities,  including  four 
designators.  Recommendations  for  future  research  include  variation  of  additional 
parameters  in  future  designs  of  experiments,  and  focused  analysis  on  individual 
community  forecasts,  independently  or  in  conjunction  with  other  communities  or 
the  overall  officer  inventory.  A  weighted  approach  to  such  an  analysis  can  require 
satisfaction  of  absolute  requirements  while  observing  flexibility  in  the  officer 
inventory  distribution  according  to  anticipated  behavior. 


XVI 


ACKNOWLEDGMENTS 


I  would  like  to  thank  Dr.  Horne  for  guiding  me  in  my  pursuit  of  analytical 
research  in  the  field  of  Navy  manpower.  Dr.  Horne's  perpetual  and  infectious 
enthusiasm,  as  well  as  his  unwavering  support  in  the  face  of  challenges,  enabled 
me  to  select  a  thesis  topic  and  conduct  research  that  has  real  value  to  the  Navy 
and  to  the  Human  Resources  Community.  I  would  like  to  thank  MAJ  Seagren  for 
his  analytical  guidance,  and  editorial  acumen,  and  for  being  available  anytime  I 
needed  to  brainstorm  or  discuss  quick  questions.  Each  of  you  contributed  to  my 
work  in  varying  ways,  and  I  am  lucky  to  have  you  both  as  my  advisors.  I  would 
like  to  thank  OPNAV  N14  for  support  and  funding,  and  in  particular  Paul  Soutter 
for  teaching  me  everything  I  needed  to  know  about  OSAM,  implementing 
changes  in  OSAM  to  assist  my  research  efforts,  and  being  constantly  available 
as  a  sounding  board  to  ensure  I  fully  understand  the  implications  of  OSAM 
forecasts.  Everyone  at  OPNAV  N14  has  been  a  joy  to  work  with  on  this  project.  I 
would  like  to  thank  Steve  Upton  for  implementing  OSAM  in  a  data  farming 
environment,  and  always  providing  a  rapid  turnaround  of  experimental  results, 
even  during  the  Christmas  holidays.  Most  importantly,  I  would  like  to  thank  my 
husband,  Paul,  for  his  moral  support  and  commitment  during  this  challenging,  but 
valuable  educational  experience. 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


I.  INTRODUCTION 


A.  PROBLEM  STATEMENT 

Manpower  and  personnel  costs  consume  a  significant  portion  of  the  U.S. 
Navy  budget  every  year,  and  Navy  planners  face  the  challenge  of  balancing 
manpower  requirements  and  mandated  end  strength  with  budget  constraints. 
The  variability  of  human  behavior  further  complicates  the  issue  of  forecasting 
strength.  The  Chief  of  Naval  Personnel’s  (NTs)  Strategic  Resourcing  Branch  is 
responsible  for  analyzing  manpower  inventory  forecasts  and  estimating  the 
Navy’s  manpower  expenditures  to  be  included  in  the  budget  and  Program 
Objectives  Memorandum  (POM)  submitted  to  the  Secretary  of  the  Navy  every 
two  years.  Forecasting  Navy  Officer  inventory  is  a  complex  problem  with  an 
imperfect  solution.  One  tool  currently  in  use  to  tackle  this  problem  is  the  Officer 
Strategic  Analysis  Model  (OSAM),  an  entity  based  model  that  follows  individual 
officers,  or  entities,  from  an  initial  inventory  through  possible  attribute  changes 
during  a  forecast  period,  resulting  in  a  projection  of  status  for  each  officer  in  the 
coming  years.  OSAM  is  not  a  tool  for  predicting  budgetary  expenses,  but  rather 
for  informing  decision  makers  as  they  consider  potential  policy  changes,  or 
analyzing  the  impact  of  policy  changes  already  in  place,  by  generating  Navy 
officer  inventory  projections  for  a  specified  time  horizon. 

In  its  current  form,  OSAM  is  complicated  to  use  and  limited  to  small-scale 
employment.  Prior  to  conduct  of  this  research,  OSAM  had  not  been  analyzed  for 
variations  over  a  large  number  of  scenarios  while  changing  multiple  input 
parameters.  The  application  of  data  farming  methodology  to  the  exploration  of 
OSAM  enhances  understanding  of  the  model’s  capabilities  and  limitations.  This 
thesis  focuses  on  identifying  effective  means  of  applying  data  farming  tools  to 
OSAM  exploration  while  targeting  specific  research  questions  related  to  the  loss 
and  lateral  transfer  functions  of  the  model. 
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B.  PURPOSE 


This  study  determines  whether  an  adaptation  of  the  OSAM  currently  in 
use  by  the  Navy’s  Strategic  Planning  and  Analysis  Directorate  (OPNAV  N14) 
could  more  effectively  inform  decisions  about  future  loss  rates  and  lateral 
transfers.  OSAM  forecasts  are  particularly  sensitive  to  changes  in  projected  loss 
rates,  and  there  is  not  an  accurate  means  to  project  loss  rates.  A  robust  design 
of  experiments  and  iterative  data  analysis  results  in  a  better  understanding  of 
how  variations  in  specific  loss  rates  and  lateral  transfer  rates  affect  OSAM 
inventory  forecasts.  Application  of  these  methods  to  OSAM  yield  a  more 
beneficial  planning  tool  for  decision  makers. 

There  is  a  great  deal  of  interest  in  retention  strategies  for  specific 
communities,  in  addition  to  officer  retention  in  the  combined  officer  corps.  Lateral 
transfers  are  essential  to  staff  communities  that  have  no  direct  accession 
sources,  and  to  retain  talented  leaders  that  otherwise  might  leave  the  Navy 
altogether.  This  study  assesses  the  sensitivity  of  OSAM  to  variations  in  loss  rate 
projections  and  lateral  transfer  rates,  and  suggests  a  systematic  approach  for 
future  researchers  to  apply  to  additional  questions. 

An  additional  product  of  this  research  is  the  modification  of  OSAM 
scenario  management  for  a  more  efficient  and  user-friendly  interface.  Automated 
model  implementation  for  data  farming  opens  up  a  wealth  of  future  research 
opportunities.  Extending  these  tools  to  examination  of  additional  aspects  of 
officer  inventory  management  may  lead  to  improved  community  management, 
accession  planning,  promotion  planning,  and  loss  rate  projections. 

C.  RESEARCH  QUESTIONS 

Navy  officer  inventory  changes  continuously,  while  long-term  planning 
relies  on  effective  personnel  forecasting  to  influence  key  decisions.  OSAM  fills  an 
essential  role  for  Navy  planners  by  modeling  the  Navy  officer  component  of 
personnel  forecasting.  In  this  model,  the  key  attributes  of  rank,  designator,  time- 
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in-service,  and  time-in-grade  describe  each  officer  in  inventory  and  adjust  as  time 
elapses.  In  addition  to  initial  inventory,  functions  describing  accession, 
promotion,  lateral  transfer,  and  loss  influence  the  total  officer  inventory 
characterization.  This  research  focuses  on  the  outcome  of  lateral  transfers  and 
losses  on  Navy  Officer  inventory  projections  by  addressing  three  specific 
research  questions: 

•  What  is  a  reasonable  range  of  loss  adjustment  factors  to  use  in 
OSAM  for  accurate  officer  inventory  projections? 

•  How  sensitive  are  officer  inventory  projections  to  varying  future  loss 
rates  for  specific  communities  or  pay  grades? 

•  Is  there  a  forecasting  benefit  to  adjusting  loss  rates  differently  for 
officers  who  applied  for  lateral  transfer  and  were  declined, 
compared  to  officers  who  never  applied  for  lateral  transfer? 

In  answering  these  research  questions,  the  application  of  specific 
hypothesis  tests  yields  insight  to  analytical  results.  These  hypotheses,  discussed 
at  length  in  Chapter  IV,  are: 

•  Primary  hypothesis  1 :  The  forecast  generated  from  experimentally 
determined  loss  adjustment  factors  is  more  accurate  than  a 
forecast  generated  from  a  set  of  loss  adjustment  factors  in  which  all 
values  are  1 .0. 

•  Primary  hypothesis  2:  The  forecast  generated  from  experimentally 
determined  loss  adjustment  factors  is  more  accurate  than  a 
forecast  generated  from  a  set  of  loss  adjustment  factors  in  which 
values  for  projection  year  one  are  equal  to  1 .0,  year  two  values  are 
1 .041 ,  year  three  values  are  1 .059,  year  four  values  are  1 .085,  and 
year  five  values  are  1.114.  These  values  describe  a  set  of  typical 
loss  adjustment  factors  that  OPNAV  N14  provided  to  model  a 
slowly  improving  economy. 
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•  Secondary  hypothesis:  Applying  the  unique  set  of  loss  adjustment 
factors  for  902x  officers,  SWOs  with  declined  lateral  transfer 
applications,  determined  from  a  rigorous  analysis  yields  a  more 
accurate  forecast  than  applying  SWO  loss  adjustment  factors  to 
these  declined  lateral  transfer  applicants. 

D.  SCOPE  AND  METHODOLOGY 

This  study  examines  Navy  officer  inventory  projection  using  the  three 
research  questions.  Automation  of  input  parameter  adjustment  and  the  post¬ 
processing  of  results  enabled  an  approach  to  the  research  questions  in  the 
context  of  data  farming.  Analysis  of  preliminary  experiments  identifies  the  loss 
rates  with  a  significant  impact  on  OSAM  forecast  accuracy  and  narrows  the 
relevant  range  of  these  loss  rates,  resulting  in  a  more  compact  design  space  for 
the  next  set  of  experiments.  Selection  of  the  final  design  space  maximizes  the 
information  obtained  about  the  model  within  the  constraints  of  available  time  and 
computing  power. 

OPNAV  N14  enhanced  OSAM  to  track  lateral  transfer  applicants  not 
selected  for  transfer,  assigning  a  distinct  loss  rate  to  these  officers.  Analysis  of 
preliminary  and  final  experiments  compares  experimental  results  to  historical 
data  to  determine  whether  the  change  to  OSAM  for  tracking  lateral  transfer 
applicants  results  in  more  accurate  officer  inventory  projections.  Additionally, 
experiments  conducted  for  various  loss  rates  were  compared  to  historical  data  to 
determine  what  range  of  loss  adjustment  factors  should  be  used  by  OSAM 
operators  in  future  scenario  consideration.  Ultimately,  this  study  demonstrates 
the  value  of  OSAM  and  its  potential  analytical  products  in  a  data  farming 
environment. 
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E.  ORGANIZATION  OF  THESIS 

This  thesis  contains  five  chapters.  Chapter  II  presents  a  brief  overview  of 
the  current  Navy  Officer  manpower  planning  process,  the  role  of  Officer  Strategic 
Analysis  Model  (OSAM)  in  personnel  policy  decisions,  and  how  data  farming  can 
multiply  the  power  of  OSAM  as  a  tool  for  Navy  decision  makers.  Chapter  II  also 
reviews  manpower  inventory  projection  related  literature.  Chapter  III  presents  the 
design  of  experiments  employed  in  data  farming  OSAM,  and  discusses  the 
methodology  of  the  data  farming  process.  Chapter  IV  presents  the  analysis  of 
experimental  results.  Chapter  V  reports  conclusions  and  recommendations  for 
future  research. 
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II.  BACKGROUND 


This  chapter  develops  the  foundation  necessary  to  understand  the 
importance  of  this  thesis  to  United  States  Navy  decision  makers.  In  addition  to 
defining  the  Navy  officer  manpower  environment  and  reviewing  past  research 
conducted  in  this  field,  the  background  contained  herein  provides  context  to  the 
methodology  and  results  detailed  in  later  chapters. 

A.  NAVY  OFFICER  MANPOWER 

A  key  component  of  budgeting  is  accurately  predicting  manpower 
inventory  in  a  dynamic  environment  subject  to  both  organizational  policies  and 
the  desires  of  individuals  within  the  system.  Officer  inventory  is  particularly 
difficult  to  predict,  as  personnel  may  enter  the  system  at  multiple  ranks,  can 
choose  to  move  between  specialties  at  numerous  points  in  a  career,  and  there  is 
no  proven  means  to  predict  officer  loss  rates. 

U.S.  Code,  Title  10  regulates  original  appointments  for  commissioned 
officers  (Section  33),  specifies  the  control  of  officers  above  the  grade  of  0-3 
(Section  32),  and  governs  aspects  of  officer  promotions,  separations,  and 
involuntary  retirements  for  all  United  States  military  branches  (Section  36).  Title 
10  codifies  some  facets  of  the  Defense  Officer  Personnel  Management  Act 
(DOPMA),  passed  into  law  in  1980.  In  addition  to  aspects  of  DOPMA  written  into 
law,  Department  of  Defense  (DoD)  policies  derived  from  Congressional  intent 
supplementing  DOPMA  legislation  provide  guidelines  on  promotion  flow  points 
and  desirable  minimum  promotion  opportunity  for  officers  in  each  pay  grade. 
(Schirmer  et  al. ,  2006) 

Officer  career  progression  occurs  upwardly  via  promotions,  and 
sometimes  laterally  via  redesignation.  Lateral  transfers  between  Navy  officer 
communities  potentially  improve  retention  and  career  satisfaction  for  individuals, 
and  enhances  the  ability  of  the  Navy  to  staff  officer  communities  properly  when 
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unexpected  shortfalls  occur.  According  to  Chief  of  Navy  Operations  (CNO) 
Instruction  (OPNAVINST  1210.5),  dated  24  Dec  2005,  the  purpose  of  lateral 
transfers  is  “to  provide  flexibility  in  the  manning  of  officer  communities.”  This 
policy  offers  flexibility  to  the  Navy,  but  also  to  individual  officers,  and  adds 
additional  uncertainty  to  the  prediction  of  loss  rates  for  applying  officers  denied 
the  opportunity  to  transfer. 

Officer  inventory  projection  models  incorporate  the  key  elements  of  officer 
career  management  and  progression  described  here.  Such  models  provide 
allowances  for  deviations  from  policies  and  practices  to  enable  evaluation  of 
policy  changes,  while  intractable  rules  integrate  laws  into  the  modeling 
environment.  A  quick  reference  guide  published  for  the  Strategic  Planning  and 
Analysis  Directorate  (OPNAV  N14)  delineates  which  aspects  of  personnel 
management  meet  law,  which  satisfy  policy,  and  which  implement  common 
practice  (Yardley  et  al. ,  2005). 

B.  OFFICER  STRATEGIC  ANALYS  IS  MODEL  (OSAM) 

1 .  Overview  of  the  Model 

OPNAV  N14  uses  OSAM  to  project  inventory  of  the  active  duty  Navy 
Officer  Corps  over  the  time  horizon  of  the  Officer  Programmed  Authorization 
(OPA).  The  OPA  specifies  how  many  officers  of  each  grade  (Ensign  through 
Captain)  are  required  in  each  designator  (73  skill  sets  modeled)  in  each 
projection  year  across  the  Fiscal  Year  Defense  Plan  (FYDP),  which  extends  six 
years  into  the  future.  OSAM  is  an  entity  based  model,  maintaining  designator, 
grade,  and  time-in-service  information  for  individual  officers,  current  and 
projected.  By  modeling  officer  behavior  consistent  with  how  the  Navy  Officer 
Corps  behaves,  OSAM  attempts  to  generate  a  supply  of  officers  with  the  right 
skills  in  the  right  grades  at  the  right  times. 

The  purpose  of  OSAM  is  to  predict  on  a  yearly  basis  the  grade,  skill,  and 
time-in-service  (or  years  of  commissioned  service)  content  of  the  Active  Navy 
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Officer  force.  In  the  model,  four  inter-related  functions  influence  expected 
outcomes:  loss  generation,  promotions,  lateral  transfers,  and  accessions.  User- 
defined  parameters  affect  each  of  these  functions,  and  they  influence  each  other 
through  interactions  within  the  model.  One  objective  of  OSAM  is  to  inform  policy 
decisions  by  generating  specific  scenarios  on  request  for  decision  makers.  For 
instance,  if  Navy  Recruiting  Command  needs  to  know  whether  their  current 
accessions  plan,  in  combination  with  NROTC  and  Naval  Academy  accessions, 
will  be  sufficient  to  supply  the  warfare  communities  with  the  required  lieutenants 
in  four  years,  OSAM  can  execute  a  scenario  with  the  given  accession  plan  to 
answer  this  question.  A  user-defined  setting  determines  whether  OSAM  runs 
constrained  to  the  inventory  set  by  OPA  for  each  year,  or  unconstrained  to 
observe  how  officer  inventory  behaves  on  its  own.  The  output  of  OSAM  is  a 
complete  inventory  of  officers  at  the  end  of  each  projection  year,  characterized 
by  the  attributes  of  grade,  designator,  date  of  rank,  time-in-service,  and  years  of 
commissioned  service. 

OSAM  has  great  potential  to  inform  policy  makers,  but  it  has  many 
weaknesses  as  well.  As  a  deterministic  model,  OSAM  provides  no  confidence 
intervals  for  the  officer  inventory  projected  by  its  unconstrained  mode. 
Furthermore,  OSAM  utilizes  four  interdependent  functions  to  generate  final 
output,  and  the  interactions  between  these  function  may  change  from  projection 
year  to  projection  year.  Of  these  four  functions,  promotions  and  accessions 
closely  reflect  Navy  policies  and  behaviors.  The  promotion  function  is  modeled 
three  different  ways,  with  user  input  determining  the  model  to  use  in  a  given 
scenario;  the  default  option  is  to  promote  to  vacancy,  as  this  is  the  underlying 
Navy  policy.  A  known  accession  plan  models  officer  gains  for  the  first  projection 
year,  and  in  subsequent  years  new  accessions  are  either  unconstrained 
(determining  according  to  the  promotions,  losses,  and  lateral  transfers)  or 
constrained  so  that  officer  end  strength  equals  OPA.  Promotion  and  accession 
functions  in  OSAM  closely  reflect  the  policies  and  practices  implemented  by 
Navy  decision  makers.  Losses  and  lateral  transfers  are  functions  that  depend 
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heavily  on  the  unpredictable  behavior  and  personal  decisions  of  individual 
officers.  The  focus  of  this  research  is  on  the  OSAM  functions  for  loss  rates  and 
lateral  transfers,  intending  to  identify  model  parameters  than  will  accurately 
capture  the  average  behavior  of  individuals. 

2.  Modeling  Officer  Loss  Rates 

OSAM  operators  at  OPNAV  N14  note  that  the  most  significant  unknown 
factor  on  model  output  is  loss  rates  for  projection  years.  Model  output  appears 
more  sensitive  to  changes  in  loss  rates  than  to  other  parameters,  yet  there  is  no 
good  prediction  system  for  officer  loss  rates  from  the  Navy  (as  opposed  to 
enlisted  loss  rates,  which  are  correlated  closely  with  unemployment  rates). 
OSAM  generates  losses  in  two  ways:  regular  loss  rates  reflect  officers  leaving 
the  Navy  by  personal  choice  or  policy  reasons,  and  forced  loss  rates  reflect  the 
practice  of  from  active  service  if  twice  non-selected  for  promotion.  For  any 
scenario,  a  user  can  also  elect  to  turn  on  or  off  the  force  out  business  rules, 
potentially  informing  decisions  on  when  and  how  often  to  allow  exceptions  to  that 
rule. 

OSAM  generates  regular  loss  rates  by  multiplying  historical  loss  rates 
(either  most  recent  year  baseline  or  a  three-year  average  from  the  preceding 
three  years)  for  a  particular  designator/rank/time-in-service  combination  by  a  loss 
adjustment  factor  defined  by  the  OSAM  user.  For  a  particular 
designator/rank/time-in-service  combination,  the  loss  adjustment  factor  for  a 
given  projection  year  is  equal  to  1.0  if  the  analyst  believes  losses  in  that 
projection  year  will  be  exactly  equal  to  historical  losses.  The  adjustment  factor 
will  be  less  than  1.0  if  the  analyst  believes  the  losses  will  be  less  than  historical 
losses,  and  greater  than  1.0  if  the  expected  losses  are  greater  than  historical 
losses.  Without  a  clear  prediction  formula  for  officer  loss  rates  in  future  years,  a 
working  set  of  parameters  for  hypothesis  testing  is  to  apply  the  same  loss 
adjustment  factor  universally  to  all  designators  and  grades  in  a  given  projection 
year  (i.e.,  loss  adjustment  factor  =  1 .041  in  year  two,  1 .059  in  year  three,  1 .085  in 
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year  four,  1.114  in  year  five).  These  values  describe  a  set  of  typical  loss 
adjustment  factors  that  OPNAV  N14  provided  to  model  a  slowly  improving 
economy. 

Each  scenario  implemented  in  OSAM  has  2,025  loss  adjustment  factors 
as  inputs  to  the  model.  Prior  to  this  research,  common  practice  applied  the  same 
loss  adjustment  factor  to  all  officers  in  a  given  projection  year,  when  in  fact  the 
loss  rates  of  pilots  five  years  from  now  may  have  changed  differently  than  the 
loss  rates  of  intelligence  officers.  While  the  loss  adjustment  factors  used  are 
reasonable  values  based  on  the  experience  of  OSAM  designers  and  users,  it  is 
desirable  to  know  just  how  sensitive  the  model  is  to  these  changes.  Data  farming 
provides  an  opportunity  to  observe  a  variety  of  loss  rates  in  a  range,  and  to 
consider  how  varying  loss  rates  differently  for  individual  designator  and  grade 
combinations  could  affect  total  officer  inventory  and  individual  community 
inventory. 

With  access  to  the  actual  Navy  officer  inventory  data  for  every  fiscal  year 
since  1978,  it  is  feasible  to  select  multiple  start  years,  and  use  OSAM  to  project 
FYDP  inventory  with  various  loss  adjustment  factors,  then  compare  projections  to 
the  actual  historical  inventories  in  projected  years.  Rather  than  varying  all  2,025 
loss  rates,  this  research  focuses  on  surface  warfare  officers  (SWOs)  and  human 
resources  (HR)  officers. 

A  third  designator,  902x,  created  in  OSAM  to  support  this  research, 

represents  SWOs  declined  an  opportunity  to  lateral  transfer.  Past  research  has 

shown  that  these  officers  have  a  significantly  higher  probability  of  leaving  the 

Navy  than  SWOs  who  successfully  lateral  transfer  or  never  apply  for  a  lateral 

transfer  (Kleyman  &  Parcell,  2010).  Each  officer  modeled  in  OSAM  belongs  to 

only  one  community  at  a  time.  In  reality,  902x  officers  as  defined  for  this  research 

are  a  subset  of  the  SWO  community,  but  in  OSAM  these  two  groups  are  disjoint 

sets.  Even  limiting  variation  of  loss  adjustment  factors  to  these  three  categories, 

each  design  point  includes  90  variables  representing  a  unique  loss  adjustment 

factor  in  OSAM.  A  preliminary  design  of  experiments  determines  which  loss  rates 
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are  most  useful  to  vary,  and  subsequent  iterations  further  narrow  the  range  of 
loss  rates  appropriate  to  use  in  forecasting  officer  inventory  over  multiple  start 
years. 


3.  Modeling  Lateral  Transfers 

There  are  three  types  of  lateral  transfers  modeled  in  OSAM:  training 
attrites,  option  officers,  and  lateral  transfers.  Training  attrites  are  officers  who  do 
not  complete  initial  training  in  their  original  designator.  Option  officers  enter  the 
Navy  with  a  contract  specifying  a  lateral  transfer  to  occur  after  completing 
surface  warfare  officer  qualification.  The  transfer  behavior  of  these  two  groups  is 
explicitly  modeled  in  a  user  defined  input  file;  an  OSAM  user  can  specify  how 
officers  move  among  designators  via  the  lateral  transfer  application  process,  but 
the  typical  user  setting  is  to  leave  lateral  transfer  behavior  unconstrained,  letting 
OSAM  calculate  the  correct  number.  In  this  case,  OSAM  determines  the  number 
of  officers  required  to  supply  lateral  transfers  to  recipient  communities.  Each 
supplying  community  supplies  a  fraction  of  all  its  officers  eligible  for  lateral 
transfer  such  that  each  community  supplies  a  similar  proportion  of  its  whole. 
Notably,  there  is  no  restriction  modeled  in  OSAM  on  the  number  of  officers  taken 
from  each  supplying  community,  which  could  potentially  leave  one  or  more  of 
these  supplying  communities  short  of  senior  officers  in  a  future  projection  year. 

A  2010  study  of  the  lateral  transfer  application  process  by  Center  for 
Naval  Analyses  (CNA)  observes  that  41%  of  applicants  disapproved  for  lateral 
transfer  leave  the  Navy  within  36  months,  while  only  10%  of  applicants  approved 
for  lateral  transfer  leave  the  Navy  within  36  months.  (Kleyman  &  Parcell) 
Currently,  OSAM  does  not  model  the  lateral  transfer  application  process  itself, 
only  the  change  in  designator  for  the  officers  approved  to  transfer.  The  findings 
of  CNA  could  have  a  significant  impact  on  the  total  Navy  officer  inventory  and  in 
particular  on  specific  officer  communities.  Kleyman  and  Parcell’s  (2010)  study 
consider  the  applicants  to  lateral  transfer  boards  in  the  period  between  2005  and 
2010.  If  applied  to  an  improved  OSAM  model,  their  findings  enable  comparison 
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of  OSAM  forecasts  for  multiple  start  years  to  the  original  OSAM  model 
projections,  and  to  the  actual  observed  historical  inventories. 

4.  Goals  of  the  Model 

“All  models  are  wrong,  but  some  are  useful”  (Box  &  Draper,  1987).  OSAM 
is  a  complex  model  used  to  observe  the  results  of  “What  if?”  scenarios  for  Navy 
planners.  At  this  time,  OSAM  is  a  Microsoft  Visual  FoxPro  9.0  executable 
application,  dependent  on  input  contained  in  60  database  files.  A  baseline 
scenario  draws  information  from  each  of  these  files,  and  adjustments  to  the 
baseline  scenarios  occur  via  individually  editing  one  or  more  of  the  60  input  files. 
A  Microsoft  Word  document  maintained  with  each  set  of  database  files  tracks 
specific  parameters  defined  in  the  database  files.  Generating  a  new  scenario 
involves  meticulous  attention  to  detail  in  adjusting  the  database  files  and 
documenting  the  changes.  Some  methods  employed  to  reduce  human  error 
when  using  this  complex  scenario  management  system  are  repetition  and 
consistent  practices.  There  is  a  single  employee  at  OPNAV  N14  whose  primary 
job  is  to  track  and  maintain  OSAM  scenarios,  and  to  run  new  scenarios  as 
needed.  Even  this  experienced  individual  requires  fifteen  to  twenty  minutes  to 
prepare  and  document  a  well-defined  scenario.  The  run  itself  takes  3-12 
minutes,  depending  on  the  machine  resources  available.  Output  analysis, 
primarily  visualization  in  pivot  charts,  occurs  immediately  after  running  a  scenario 
and  takes  15-30  minutes,  depending  on  the  number  and  types  of  questions  of 
interest.  Despite  complexities  in  implementation,  OSAM  is  a  powerful  tool  with 
great  potential  for  informing  Navy  leadership  on  the  impacts  of  proposed  policy 
changes. 

This  thesis,  in  conjunction  with  the  Simulation  Experiments  &  Efficient 
Designs  (SEED)  Center  for  Data  Farming,  considers  different  choices  of  input 
parameters  to  OSAM  within  reasonable  ranges,  performs  sensitivity  analysis  on 
these  parameters,  and  suggests  reasonable  values  or  ranges  of  values  to  use  as 
inputs  to  OSAM.  Preliminary  analysis  determines  reasonable  ranges  of  variation 
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for  ninety  loss  adjustment  factors.  The  input  to  OSAM  that  has  the  greatest 
influence  on  results  is  loss  rates,  for  which  there  is  no  reliable  prediction  means 
at  this  time.  The  ability  to  execute  numerous  runs  of  OSAM  in  a  short  period  in  a 
data  farming  environment  provides  an  opportunity  to  understand  the  impact  of 
varying  loss  rates  on  officer  inventory. 

C.  LITERATURE  REVIEW 

For  as  long  as  there  have  been  organizations,  managers  have  had  an 
interest  in  predicting  the  flow  of  manpower  into,  within,  and  out  of  them;  military 
leaders  have  faced  this  challenge  for  as  long  as  armies  have  been  around.  This 
need  for  leaders  to  plan  ahead  evolved  over  time  to  simple  manpower  modeling 
and  then  to  multifaceted  projection  planning  as  the  size  and  complexity  of 
organizations  expanded  over  the  centuries.  Manpower  modeling  in  general  and 
military  manpower  modeling  in  particular  has  been  the  subject  of  targeted 
operations  research  since  before  operations  research  has  been  a  recognized 
discipline.  As  availability  of  computing  power  has  improved  dramatically, 
manpower  modeling  has  reached  new  levels  of  complexity;  the  research 
conducted  over  the  past  few  decades  has  laid  a  solid  foundation  for  the  Officer 
Strategic  Analysis  Model  examined  in  this  thesis. 

A  review  of  literature  previously  published  on  the  subject  of  manpower 
modeling  provides  background  information  on  the  subject,  identifies  successful 
model  explorations,  and  highlights  limitations  of  previous  studies  in  the  context  of 
this  thesis.  Publications  on  the  subject  of  manpower  modeling  are  numerous,  but 
none  apply  data  farming  tools  to  exploring  an  existing  officer  manpower  model. 
This  literature  review  identifies  research  on  similar  or  related  topics,  such  as 
developing  new  officer  inventory  models,  or  data  farming  an  existing  enlisted 
manpower  model.  Despite  the  significant  gap  between  this  thesis  and  previously 
published  works,  each  of  the  sources  identified  in  this  literature  review  provide 
background  and  context  to  the  evolving  field  of  forecasting  military  personnel 
accurately. 
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The  results  of  this  literature  review  cover  two  distinct  topics:  Navy 
manpower  modeling,  and  other  service  manpower  modeling.  The  discussion  on 
Navy  manpower  modeling  falls  under  the  subdivisions  of  inventory  projection 
research  and  studies  targeting  lateral  transfer  processes  and  loss  rate 
projections. 

1.  Navy  Manpower  Modeling 

a.  Inventory  Projection 

One  definition  of  manpower  planning  is  the  interaction  of  three 
processes:  predicting  future  demand,  predicting  future  supply,  and  evaluating 
policies  intended  to  bring  predicted  demand  and  predicted  supply  as  close 
together  as  possible  (Edwards,  1983).  Inventory  projection  brings  together  all 
three  of  these  functions  by  using  organizational  knowledge  and  policies  to  predict 
supply  and  demand,  and  observing  the  gap  between  them.  Each  inventory 
projection  model  considered  in  this  literature  review  has  the  ultimate  goal  of 
informing  decision  makers  on  the  impact  of  potential  policy  changes. 

Clark  (2009)  develops  a  linear  optimization  program, 
Requirements-Driven  Cost-Based  Manpower  Optimization  (RCMOP),  to  project 
monthly  values  for  Navy  officer  inventory  by  minimizing  unmet  manpower 
requirements  without  over-executing  the  budget.  Like  OSAM,  Clark  (2009) 
models  officer  inventory  as  a  function  of  four  component  functions:  promotions, 
accessions,  lateral  transfers,  and  losses.  RCMOP  assigns  penalties  to  unmet 
manpower  requirements  to  determine  priority  in  executing  the  budget,  while 
OSAM  can  either  constrain  inventory  to  meet  manpower  requirements  or  allow 
the  officer  inventory  to  progress  naturally  according  to  expected  behavior  of 
individuals  if  unconstrained  by  OPA. 

Wheeler  (2010)  builds  on  Clark’s  2009  RCMOP  model.  Wheeler 
considers  variation  of  loss  rates  and  additional  officer  communities,  as  well  as 
increasing  the  time  horizon  considered.  The  majority  of  officer  communities 
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remain  in  the  “other”  category  under  Wheeler’s  analysis.  The  nature  of  OSAM  as 
a  simulation  model  allows  incorporation  of  each  Navy  officer  community 
separately  to  obtain  high-resolution  inventory.  Clark  (2009)  and  Wheeler  (2010) 
focus  on  meeting  budget  requirements,  a  goal  distinctly  different  from  that  of  this 
research,  which  focuses  on  assessing  the  treatment  of  loss  rates  employed  in 
generating  an  accurate  forecast. 

b.  Lateral  Transfer  Modeling  or  Analysis 

In  1997,  the  Center  for  Navy  Analyses  (CNA)  examines  the  Navy’s 
lateral  transfer  system.  Moore  and  Reese  (1997)  assess  how  the  policy  of 
staffing  Restricted  Line  (RL)  and  Staff  Corps  communities  fit  into  a  strategy  to 
increase  retention  and  career  satisfaction  among  officer.  At  the  time  of  Moore 
and  Reese’s  study,  there  is  a  shortage  of  lateral  transfers  to  sustain  the  RL  and 
Staff  communities;  a  secondary  goal  of  their  research  is  to  determine  whether  an 
adjustment  to  transfer  rates  is  appropriate.  This  CNA  study  observes  that  lateral 
transfer  arising  from  training  attrition  early  in  an  officer’s  career  fail  to  obtain 
warfare  qualification  more  often  than  officers  originally  placed  in  a  designator  do; 
Moore  and  Reese  suggest  that  this  is  due  to  a  mismatch  between  the  job  and  the 
officer.  The  observations  in  this  1997  analysis  lay  the  foundation  for  numerous 
studies  on  the  subject  of  lateral  transfers  in  subsequent  years,  but  do  not  attempt 
to  model  lateral  transfers  in  a  forecasting  environment. 

In  2004,  Monroe  and  Cymrot  analyze  a  proposed  policy  of  cutting 
Navy  officer  accessions  and  limiting  lateral  transfers,  ultimately  determining  that 
reducing  Surface  Warfare  Officer  (SWO)  accessions  by  160  officers  each  year 
could  save  the  Navy  $91  million.  This  study  considers  only  the  SWO  community, 
using  productivity  measures  to  estimate  the  tradeoffs  between  keeping  officers  in 
the  SWO  community  and  allowing  them  to  transfer  in  greater  numbers  to  RL  or 
Staff  communities.  Generally,  any  SWO  queried  in  an  unofficial  context  will  state 
that  fellow  SWOs  pursuing  a  lateral  transfer  and  declined  the  opportunity  to 
transfer  often  elect  to  leave  the  Navy,  thus  still  creating  a  gap  in  SWO 
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manpower.  Monroe  and  Cymrot  (2004)  take  steps  to  quantify  these  choices  by 
SWOs,  and  lay  the  groundwork  for  future  studies  to  drill  down  further.  Monroe 
and  Cymrot  do  not  address  the  questions  undertaken  in  this  thesis,  but 
nonetheless  provide  background  important  to  understanding  and  interpreting 
forecast  results. 

In  2007,  Ryan  identifies  factors  that  lead  Unrestricted  Line  (URL) 
officers  to  request  lateral  transfers,  identifies  lateral  transfer  selection  criteria 
used  by  selection  boards,  and  determines  through  regression  analysis  that 
officers  turned  down  by  a  lateral  transfer  selection  board  have  a  different 
retention  likelihood  than  officers  who  apply  and  are  selected.  “Officers  who  apply 
for  lateral  transfer  but  are  not  selected  are  more  than  twice  as  likely  to  leave  the 
Navy  as  those  who  are  selected”  (Ryan,  2007,  p.  73).  Ryan’s  research  does  not 
quantify  the  impact  of  his  findings  on  inventory  projections,  as  this  thesis  begins 
to  do. 

Building  on  Ryan’s  2007  work,  Kleyman  and  Parcell  (2010)  conduct 
a  thorough  statistical  analysis  of  lateral  transfer  applicants  over  a  five-year 
period,  and  determine  a  lower  bound  impact  of  denying  applicants  solely  based 
on  supplying  community  quotas.  Ryan  (2007)  and  Kleyman  and  Parcell  (2010) 
note  retention  differences,  but  stop  short  of  applying  these  differences  to 
modeling  loss  rates  for  lateral  transfer  applicants,  a  task  uniquely  suited  to  data 
farming  an  inventory  projection  model  such  as  OSAM. 

2.  Other  Service  Manpower  Modeling 

In  1983,  a  simulation  model  for  military  personnel  analysis,  the  Accession 
Supply  Costing  and  Requirements  (ASCAR)  model,  develops  to  compute 
projected  shortfalls  in  desired  end  strength  and  total  man-years  (Collins  et  al ., 
1983).  This  model,  like  OSAM,  informs  decision  makers  on  the  impact  of 
potential  personnel  policies.  This  model  is  an  important  predecessor  to  OSAM  in 
both  its  design  and  its  purpose,  but  as  a  model  for  active  duty  enlisted  personnel, 
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ASCAR  does  not  consider  the  complexities  of  prior  service  and  promotions  in 
controlled  grades,  which  this  thesis  must  incorporate  in  the  analysis  of  officer 
inventory  projections. 

In  1991,  Grant  publishes  a  special  report  discussing  a  non-line  officer 
projection  model  developed  for  the  United  States  Air  Force  to  inform  the  effect  of 
policy  decisions,  such  as  compensation  and  promotion  adjustments,  on  future 
force  structure.  This  model  is  an  discrete  event  simulation  and  includes  many  of 
the  functional  aspects  that  OSAM  addresses,  though  there  is  no  allowance  for 
lateral  transfers  in  this  Air  Force  model.  Notably,  the  non-line  officer  projection 
model  presented  by  Grant  (1991)  does  not  consider  the  non-line  officer  force  as 
an  aggregate.  That  is,  the  model  provides  a  forecast  for  each  officer  community 
in  isolation,  thus  missing  any  interaction  effects  that  affect  the  total  force. 

Fiebrandt  (1993)  develops  and  implements  a  U.S.  Coast  Guard  Rating 
Forecast  Model  to  project  inventory  and  personnel  flow  by  rating.  Though  this 
model  is  for  enlisted  personnel  and  for  a  different  military  service,  it  treats 
functions  similarly  to  OSAM,  though  the  constraints  behave  differently.  The 
greatest  shortcoming  of  Fiebrandt’s  model  in  the  context  of  officer  inventory 
projection  is  that,  like  Grant  (1991),  it  models  only  one  community  at  a  time, 
losing  the  information  gained  from  observing  redesignations  between  ratings. 

In  2002,  Schrews  develops  a  new  optimization  model  for  enlisted 
manpower  projection  in  the  United  States  Army  Active  Guard  Reserve  (AGR). 
The  model  developed  by  Schrews  (2002)  tracks  soldiers  through  a  simulation 
with  the  same  attributes  that  OSAM  follows,  and  deals  with  accessions  at  pay 
grades  E-4  to  E-9,  rather  than  at  the  initial  training  point.  Though  Schrews  (2002) 
does  address  loss  rate  variations,  these  variations  have  a  small  impact  on 
results,  as  new  gains  to  the  system  replacing  losses  require  only  a  one-week 
training  pipeline.  OSAM  must  incorporate  training  pipelines  from  6  months  to  8 
years  for  different  designators,  so  the  effect  of  loss  rates  on  forecasts  warrants 
rigorous  analysis. 
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Erdman  (2010)  uses  experimental  design  and  data  analysis  to  study  the 
U.S.  Army’s  Enlisted  Specialty  (ES)  model.  The  ES  model  uses  coefficients  to 
set  penalties  and  rewards  on  different  components  of  its  objective  function;  the 
user  can  adjust  these  coefficients  to  focus  the  optimization  on  certain  goals  to 
observe  the  impact  on  future  decisions.  Erdman’s  2010  thesis  uses  data  farming 
to  select  the  coefficient  values  that  had  the  greatest  impact  on  lowering  the 
deviation  between  inventory  and  authorizations  over  the  planning  horizon.  Like 
OSAM,  the  ES  model  projects  manpower  inventory  in  specific  job  types  and 
ranks.  While  the  modeling  methods  employed  are  very  different  between  the  two 
(entity-based  simulation  in  OSAM  vs.  linear  optimization  in  ES  model),  the 
applicability  of  data  farming  is  similar  for  each.  The  use  of  historical  data  as  a 
representation  of  the  future  to  assess  input  parameter  values  is  important  to  this 
thesis,  as  it  was  in  assessing  the  ES  model. 

While  the  studies  reviewed  in  this  chapter  each  provide  insight  to  the 
intricacies  of  Navy  officer  manpower  modeling,  none  of  them  addressed  the 
issue  of  modeling  all  officers  in  a  service,  accounting  for  individual  communities 
and  transition  between  them.  OSAM’s  capability  to  handle  these  complexities  is 
unique.  With  this  fact  in  mind,  the  application  of  an  effective  design  of 
experiments,  data  farming,  and  simulation  analysis  to  OSAM  is  the  subject  of  the 
remaining  chapters  in  this  thesis. 
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III.  METHODOLOGY  AND  APPROACH 


This  chapter  introduces  the  tools  and  methods  employed  in  gathering, 
processing,  and  analyzing  data  from  495  OSAM  simulations.  Each  simulation 
produces  about  80,000  rows  of  data,  each  of  which  represents  a  Navy  officer  in  a 
given  year.  The  selection  of  experimental  designs  and  the  reduction  of  generated 
data  to  a  single  measure  of  effectiveness  presents  a  unique  challenge.  This 
chapter  discusses  analytical  methods  used  is  assessing  the  forecast  accuracy  of 
OSAM. 

A.  DATA  FARMING 

1.  Definition  and  Application 

Data  farming  is  a  continually  evolving  combination  of  methods  that 
capitalizes  on  high  performance  computing  to  explore  a  decision  space  more 
completely  than  traditional  analytics  typically  allow.  The  opportunity  to  observe 
an  entire  landscape  of  solutions  enables  analysts  to  interpret  results,  assess 
model  validity,  identify  and  examine  outliers,  and  explore  otherwise 
insurmountable  research  questions  (Horne  &  Meyer,  2010). 

Any  model  with  multiple  input  parameters  subject  to  variation  could  be  a 
candidate  for  data  farming.  Efficient  experimental  design  combined  with 
automation  of  input  parameter  variations  can  leverage  high  performance 
computing  to  explore  a  large  design  space  (Horne  &  Schwierz,  2008).  Figure  1 
depicts  the  iterative  nature  of  data  farming.  The  scenario  building  loop  in  Figure  1 
includes  building  a  model  and  defining  research  questions  to  explore  with  that 
model.  This  research  uses  a  pre-built  model,  OSAM,  so  this  scenario  building 
loop  refers  to  the  specification  of  research  questions  and  determination  of  how  to 
proceed  with  model  exploration.  The  scenario  run  space  execution  loop  on  the 
right  hand  side  of  Figure  1  describes  the  process  of  designating  a  design  space, 
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executing  multiple  experiments,  analyzing  and  interpreting  results,  and  repeating 
the  process  as  appropriate  (Horne  &  Meyer,  2004). 


Data  Farming  Loop 

Figure  1 .  Iterative  nature  of  data  farming  (From  Horne  &  Meyer,  2004). 

A  key  component  of  effective  data  farming  is  efficient  experimental  design. 
Design  of  experiments  (DOE)  is  a  field  that  applies  a  systematic  approach  to 
observing  the  impact  of  changing  variables  in  a  simulation.  The  application  of 
DOE  within  the  data  farming  loop  in  Figure  1  enables  observation  of  single 
variable  effects  and  interactions  between  variables.  The  Appendix  summarizes 
the  components  constructed  to  implement  OSAM  in  a  data  farming  environment. 

2.  Design  of  Experiments 

Data  farming  enables  study  of  a  model’s  response  to  multiple  variations  of 
many  input  parameters,  available  computing  power  and  experimental  design 
features  magnify  the  benefits  of  data  farming  techniques.  As  one  objective  of 
this  thesis  is  to  explore  the  applicability  of  data  farming  to  OSAM,  the  scope  of 
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this  experiment  is  limited  to  varying  90  parameters,  as  described  in  Tables  1-5, 
describing  the  loss  adjustment  factors  of  the  Surface  Warfare  and  Human 
Resources  communities.  All  DOEs  retain  the  mapping  described  in  these  tables, 
though  in  later  experiments  many  input  parameters  remain  constant  at 
analytically  determined  optimal  values.  To  assist  in  exploring  the  lateral  transfer 
research  question,  analysts  at  OPNAV  N14  adapted  OSAM  to  add  a  new 
designator,  902x,  to  be  defined  as  needed  for  specific  research  purposes.  For 
this  investigation,  the  902x  designator  includes  SWOs  declined  the  opportunity  to 
lateral  transfer.  The  902x  designator  has  its  own  set  of  historical  loss  rates, 
distinct  from  SWOs  who  never  applied  for  lateral  transfer. 


Table  1 .  Each  input  parameter  to  vary  in  OSAM  simulation  mapping  to  its 
description  of  attributes.  This  table  shows  the  mapping  for  the  loss 
adjustment  factors  in  the  first  projection  year  of  each  simulation. 


loss adj  factor  mapping 

Projection  Year 

designator 

grade 

input  parameter 

1 

VI 

2 

V2 

HR  (1200) 

3 

V3 

4 

V4 

5 

V5 

6 

V6 

1 

Ml 

SWO  (1110) 

2 

V8 

1 

&  SWO 

3 

V9 

Trainee 

4 

V10 

(1160) 

5 

Vll 

6 

V12 

SWOs 

declined  for 

Lateral 

Transfer 

(902x) 

1 

V13 

2 

V14 

3 

V15 

4 

V16 

5 

Mil 

6 

V18 
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Table  2. 


Mapping  for  the  loss  adjustment  factors  in  the  second  projection  year 
of  each  simulation. 


loss_adj  factor  mapping 

Projection  Year 

designator 

grade 

input  parameter 

2 

HR  (1200) 

1 

V19 

2 

V20 

3 

V21 

4 

V22 

5 

V23 

6 

V24 

SWO  (1110) 
&  SWO 

Trainee 

(1160) 

1 

V25 

2 

V26 

3 

V27 

4 

V28 

5 

V29 

6 

V30 

SWOs 

declined  for 

Lateral 

Transfer 

(902x) 

1 

V31 

2 

V32 

3 

V33 

4 

V34 

5 

V35 

6 

V36 

Table  3.  Mapping  for  the  loss  adjustment  factors  in  the  third  projection  year. 


loss_adj  factor  mapping 

Projection  Year 

designator 

grade 

input  parameter 

1 

V37 

2 

V38 

HR  (1200) 

3 

V39 

4 

V40 

5 

V41 

6 

V42 

1 

V43 

SWO  (1110) 

2 

V44 

3 

&  SWO 

3 

V45 

Trainee 

4 

V46 

(1160) 

5 

V47 

6 

V48 

SWOs 

declined  for 

Lateral 

Transfer 

(902x) 

1 

V49 

2 

V50 

3 

V51 

4 

V52 

5 

V53 

6 

V54 

24 


Table  4. 


Mapping  for  the  loss  adjustment  factors  in  the  fourth  projection  year. 


loss_adj  factor  mapping 

Projection  Year 

designator 

grade 

input  parameter 

4 

HR  (1200) 

1 

V55 

2 

V56 

3 

V57 

4 

V58 

5 

V59 

6 

V60 

SWO  (1110) 

&SWO 

Trainee 

(1160) 

1 

V61 

2 

V62 

3 

V63 

4 

V64 

5 

V65 

6 

V66 

SWOs 

declined  for 

Lateral 

Transfer 

(902x) 

1 

V67 

2 

V68 

3 

V69 

4 

V70 

5 

V71 

6 

V72 

Table  5.  Mapping  for  the  loss  adjustment  factors  in  the  fifth  projection  year. 


loss_adj  factor  mapping 

Projection  Year 

designator 

grade 

input  parameter 

1 

V73 

2 

V74 

HR  (1200) 

3 

V75 

4 

V76 

5 

V77 

6 

V78 

1 

V79 

SWO  (1110) 

2 

V80 

5 

&SWO 

3 

V81 

Trainee 

4 

V82 

(1160) 

5 

V83 

6 

V84 

SWOs 

declined  for 

Lateral 

Transfer 

(902x) 

1 

V85 

2 

V86 

3 

V87 

4 

V88 

5 

V89 

6 

V90 

25 


In  a  simulation  environment  with  a  small  number  of  variable  input 
parameters  and  a  discrete  number  of  levels  for  each  parameter,  it  is  possible  to 
run  an  experiment  covering  every  combination  of  variables.  This  set  of  conditions 
describes  a  full  factorial  design,  and  this  type  of  experiment  exhausts  a  design 
space,  providing  maximum  information  to  an  analyst.  Unfortunately,  it  is 
infeasible  to  conduct  a  full  factorial  experiment  on  most  real  world  models,  and 
impossible  for  many,  due  to  the  exponential  growth  of  design  points  required  with 
the  addition  of  each  factor  or  factor  level.  Methods  in  the  area  called  DOE  devote 
resources  to  identifying  and  exploring  experimental  designs  that  cover  much  of 
the  design  space  with  a  fraction  of  the  design  points  that  would  appear  in  a  full 
factorial  design.  Latin  hypercubes  (LHs)  and  nearly  orthogonal  Latin  hypercubes 
(NOLHs)  are  two  design  types  that  satisfy  this  challenge  neatly  (Sanchez,  2006). 

When  the  factors,  or  variables,  of  a  DOE  are  columns  with  N  rows 
representing  the  possible  levels  for  each  factor,  a  randomized  permutation  of 
each  column  results  in  a  Latin  hypercube  design.  This  type  of  design  is  most 
beneficial  in  experiments  with  large  numbers  of  factors,  and  a  randomized  LH 
has  good  orthogonality  properties  when  the  number  of  design  points  is  much 
greater  than  the  number  of  factors.  For  smaller  designs,  it  is  important  to  select  a 
LH  design  with  low  pairwise  correlation  between  columns.  An  alternative  to 
generating  multiple  LHs  and  choosing  one  with  low  pairwise  correlations  is  to  use 
an  NOLH  design.  The  SEED  Center  at  NPS  maintains  spreadsheets  pre-built  to 
generate  NOLH  designs  for  small  and  moderate  numbers  of  factors.  Even  with  a 
small  number  of  factors  to  vary,  a  full  factorial  design  is  cumbersome  with  more 
than  two  levels.  An  NOLH  design  has  good  space-filling  and  orthogonality 
properties,  does  not  need  checking  for  pairwise  correlations,  and  requires  a 
fraction  of  the  design  points  a  full  factorial  design  would  need  (Sanchez,  20006). 

Running  a  large  number  of  simulations,  one  for  each  design  point, 
generates  massive  amounts  of  data.  A  relationship  exists  between  the 
explanatory  variables,  or  input  parameters,  and  one  or  more  response  variables; 
determining  this  relationship  correctly  is  often  the  goal  of  simulation  analysis.  In 
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1951,  Box  and  Wilson  introduce  response  surface  methodology  as  a  means  to 
approximate  this  relationship  using  a  second-degree  polynomial.  This 
methodology  has  proven  effective  and  easy  to  apply  to  simulation  analysis,  even 
when  details  about  the  simulation  process  used  to  generate  the  data  are 
unknown.  Using  current  computing  and  available  analytical  software,  a 
regression  model  allows  all  input  parameters  to  enter  the  model  as  independent 
variables,  along  with  the  quadratic  terms  for  each  input  parameter,  and  all 
possible  two-way  interactions  between  the  input  parameters.  For  an  experiment 
with  90  input  parameters,  this  methodology  generates  a  regression  model  with 
an  unwieldy  number  of  terms,  and  additional  statistical  tools  serve  to  simplify  the 
model.  A  simulation  analysis  conducted  after  each  DOE  results  in  reduction  of 
input  parameters  to  include  in  the  subsequent  DOE  if  some  parameters  do  not 
appear  in  the  response  surface  model,  or  optimization  of  all  generated  models 
yields  the  same  solution  for  some  parameters. 

In  conducting  this  research,  the  first  DOE  varies  108  factors  over  108 
design  points,  and  is  a  randomized  LH  design  (S-plus  script  for  Latin  hypercubes, 
2011).  Each  factor  is  a  continuous  variable,  but  the  design  generation  uses  108 
discrete  levels  for  these  factors.  This  first  DOE  intends  to  analyze  variation  over 
a  six  year  forecast  period,  but  base  year  data  files  for  OSAM  were  only  available 
as  far  back  as  2007,  so  the  simulation  analysis  was  limited  to  a  five  year 
projection  period.  The  variable  mapping  described  in  Tables  2-5  shows  only  the 
90  factors  contributing  to  the  analysis.  Of  these  90  remaining  factors,  five 
represent  902x  officers  of  grade  01  (Ensigns).  Navy  policy  prohibits  officers  with 
less  than  two  years  of  commissioned  service  from  applying  for  lateral  transfer; 
this  policy  effectively  means  that  902x  Ensigns  never  appear  in  the  model.  The 
inclusion  of  these  five  factors  in  the  first  DOE  is  an  oversight,  and  subsequent 
DOEs  do  not  consider  them. 

The  second  DOE  varies  65  factors  over  128  design  points,  and  is  an 
NOLH  design  (Vieira,  Sanchez,  Kienitz,  &  Belderrain,  2011).  The  third  DOE 
varies  29  factors  over  128  design  points,  and  is  an  NOLH  design  (Sanchez, 
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2005).  This  third  DOE  runs  first  with  all  902x  parameters  equal  to  their 
corresponding  SWO  parameters,  and  a  second  time  with  all  902x  parameters 
equal  to  values  calculated  in  earlier  analyses.  Conventionally,  as  the  number  of 
factors  in  a  DOE  decreases,  so  does  the  number  of  design  points  needed.  This 
research  used  fewer  design  points  in  earlier  DOEs  to  explore  and  analyze  the 
model  within  a  reasonable  period.  All  DOEs  execute  five  times,  once  for  each  of 
the  five  base  years,  2007-201 1 ,  that  generate  a  forecast  comparable  to  available 
historical  data. 


B.  DATA  HANDLING 


Every  design  point  generates  about  80,000  rows  of  data,  each 
representing  a  single  officer  in  one  projection  year  of  the  forecast  period.  To 
conduct  a  simulation  analysis,  it  is  necessary  to  condense  this  data  into  a  single 
measure  of  effectiveness  for  each  design  point.  The  primary  research  focus  is  to 
assess  the  loss  adjustments  factors’  impact  on  forecast  accuracy,  so  the  analysis 
compares  forecast  inventory  to  actual  inventory  for  each  year  that  historical  data 
is  available.  Relative  difference,  absolute  difference,  or  mean  squared  error  can 
all  be  effective  measures  of  forecast  accuracy,  but  each  of  these  needs  to  be 
weighted  proportionately  when  condensing  so  much  data  into  a  single  value.  A 
straightforward  solution  is  to  calculate  the  mean  absolute  proportional  error 
(MAPE)  for  each  design  point. 


I 


mape,„,,= 


actual state  -  forecast n 

actualsme 

N 


where  N  is  the  number  of  data  points  included. 


In  the  context  of  this  thesis,  the  state  denoted  by  subscript  refers  to  the 
combination  of  skill  (SK),  grade  (GR),  and  fiscal  year  (FY).  To  calculate  MAPE 
for  each  design  point  requires  significant  data  manipulation.  First,  culling  the  data 
to  extract  only  SWOs,  HRs,  and  902x  officers  enables  matching  forecast 
inventory  with  actual  historical  data  for  only  the  designators  of  interest  to  this 
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thesis.  Then,  the  SWO  community  absorbs  all  902x  officers  for  comparison  with 
historical  inventory.  The  initial  redesignation  of  these  officers  to  902x  allows  them 
to  behave  uniquely  in  OSAM,  but  they  are  still  SWOs,  and  must  be  included  in 
the  total  SWO  inventory  in  evaluating  forecast  accuracy.  The  remaining  data 
includes  a  SWO  and  HR  inventory  for  each  projection  year  and  grade  for  all  90 
design  points.  This  table  joins  with  a  historical  inventory  table  to  generate 
MAPEfy  values  for  each  DP.  The  FY  subscript  associated  with  each  MAPE 
value  indicates  that  this  value  measures  forecast  accuracy  for  a  specific  fiscal 
year;  each  projection  year  OSAM  generates  yields  a  separate  MAPEfy  model.  As 
the  manipulation  and  combination  of  data  tables  proceeds,  the  number  of  rows, 
N,  describing  the  data  changes  several  times,  rendering  it  a  misleading  value  to 
include  in  final  MAPE  calculation.  Instead,  the  traditional  MAPE  formula  is 
adapted  to  normalize  for  total  forecast. 


mapefy 


i 

SK,GR 


actual SK  GR  FY  for  ecastSK  GR  FY 

- — — - •  forecast SKGRFY 

ti  L  l/t  Li  L  Q ^  FY 


Yj  forecast sp  ,GR,FY 

SK,GR 


The  resulting  data  set  includes  one  MAPEfy  value  for  each  design  point, 
used  in  the  next  phase  of  analysis  to  generate  regression  models.  Conducting 
MAPE  calculations  proportioned  on  rank  and  skill  in  this  fashion  ensures  that  a 
small  group  of  officers,  such  as  HR  Captains  (06s),  do  not  exert  the  same 
influence  on  the  overall  MAPE  model  as  a  large  subset,  such  as  SWO  Ensigns 
(01s).  This  method  simultaneously  ensures  that  the  influence  of  deviations  within 
a  subset  exert  a  proportional  influence  on  the  model  (i.e.,  a  forecast  with  five 
more  HR  Captains  than  historical  has  a  greater  impact  on  the  HR  Captain 
component  than  a  forecast  with  five  more  SWO  Ensigns  than  historical).  This 
second  attribute  is  particularly  important  if  applying  this  method  to  modeling 
MAPE  by  designator  and  fiscal  year  (MAPEfy.sk),  rather  than  only  by  fiscal  year 
(MAPEfy)  as  implemented  in  this  simulation  analysis. 
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The  last  step  in  the  data-handling  phase  of  analysis  is  subsetting  MAPE 
calculations  by  fiscal  year,  generating  a  file  for  each  projection  year  with  a  MAPE 
value  for  each  design  point.  Once  matched  with  the  original  DOE  file,  this  data 
table  is  ready  for  the  modeling  phase  of  analysis. 

C.  REGRESSION  ANALYSIS 

OSAM’s  forecast  accuracy  is  a  function  of  many  complex  inter-related 
elements,  none  of  which  exhibit  linear  relationship  in  the  execution  of  the  model. 
Design  of  experiments  and  simulation  analysis  enable  reduction  of  the  complex 
relationship  between  loss  adjustment  factors  and  forecast  accuracy  to  regression 
modeling.  Ordinary  least  squares  regression  is  insufficient  to  describe  this 
relationship,  but  a  stepwise  regression  allowing  all  terms  in  a  response  surface 
model  can  predict  MAPE  effectively.  A  response  surface  includes  each 
parameter  varied  in  the  model,  quadratic  terms  for  each  variable  and  all  two-term 
interactions  between  variables  (Box  &  Wilson,  1951). 

Using  a  minimum  Akaike’s  Information  Criterion  (AIC)  stopping  rule, 
stepwise  regression  identifies  the  best  model,  based  on  subjective  statistical 
rules,  of  MAPE  for  each  projection  year  and  base  year;  each  DOE  generates  15 
MAPE  models.  The  model  naming  convention  used  in  this  thesis  is  base  year- 
MAPE-projection  year,  so  the  model  named  2007MAPE1  is  the  MAPE  model  for 
the  first  projection  year  (FY07)  of  the  scenario  with  base  year  2007. 

For  the  first  DOE,  the  MAPE  models  include  a  large  number  of  terms  and 
exhibit  excellent  fit,  many  with  r-square  values  greater  than  0.98.  A  smaller 
number  of  terms  is  desirable,  so  this  researcher  employs  a  process  that  runs  the 
model  iteratively  to  observe  t-statistics  for  individual  model  coefficients  and 
remove  additional  terms  manually  from  the  stepwise  model.  The  modeling 
challenge  is  to  achieve  a  good  enough  fit  while  limiting  the  total  number  of  terms 
included  in  the  model.  The  solution  to  this  challenge  is  to  make  judgment  calls  in 
balancing  the  AIC  value  and  goodness  of  fit  statistics  with  the  likelihood  that 
included  terms  are  individually  significant  to  the  model.  This  balancing  act  is 
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simpler  with  later  DOEs,  when  fewer  terms  in  the  model  limit  the  goodness  of  fit, 
disallowing  the  opportunity  to  remove  terms  from  the  initial  stepwise  model. 

D.  OPTIMIZATION  BY  NONLINEAR  PROGRAMMING 

It  is  desirable  to  minimize  the  absolute  deviation  of  forecast  inventory  from 
actual  inventory,  but  a  direct  attempt  to  minimize  MAPE,  a  function  of  loss 
adjustment  factors,  will  result  in  most  or  all  factors  being  set  to  their  minimum 
allowable  values.  Further,  it  is  desirable  to  compare  multiple  MAPE  models 
simultaneously.  When  comparing  multiple  models  in  a  group,  or  meta-model,  the 
maximum  MAPE  among  these  models  is  the  value  to  minimize.  Approaching  the 
problem  in  this  manner  results  in  most  or  all  models  in  the  meta-model  ending 
with  very  similar  MAPE  values,  resulting  in  a  better  solution  set  than  could  be 
reached  by  minimizing  the  total  or  average  MAPE  of  the  meta-model.  This 
problem  formulation  is: 

min  max{MAPE) 

xeV  L  xeV 

s-t-  xmin  <  x  <  xmax  V  x  e  {V\,V2,..y90} 

MAPE  >  0  V  MAPE  e  meta-model 

The  simplest  meta-model  to  use  includes  each  of  the  fifteen  MAPE 
models  generated  for  the  DOE.  Some  base  year  scenarios  provide  MAPE  data 
for  only  one  or  two  years,  and  only  one  or  two  models  cover  some  projection 
years.  To  account  for  this  uneven  distribution  of  data  and  reduce  the  influence  of 
any  single  model  on  the  final  solution  set,  a  formulation  of  meta-models  derives 
from  three  different  grouping  categories.  One  category  includes  all  models 
generated  from  a  given  base  year  scenario.  A  second  category  includes  all 
models  that  project  a  given  projection  year,  independent  of  base  year.  A  third 
category  includes  all  models  that  predict  a  given  fiscal  year.  Table  6 
demonstrates  the  meta-model  groupings  considered  in  determining  optimal 
values  for  each  loss  adjustment  factor. 
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A  benefit  of  this  problem  formulation  is  its  ease  of  implementation  in 
Microsoft  Excel.  A  known  limitation  of  Excel’s  solver  feature  is  that  the  solution  is 
a  local  minimum,  but  not  necessarily  a  global  minimum.  The  solution  set  returned 
by  Excel  is  dependent  on  the  initial  values  selected  for  all  variables.  For  this 
reason,  the  initial  values  chosen  were  consistent  and  equal  to  1.0  for  all  loss 
adjustment  factors.  This  practice  does  not  guarantee  the  smallest  possible  MAPE 
value,  but  the  potential  errors  due  to  this  software  property  are  smoothed  by  the 
averaging  of  all  results  over  the  fifteen  meta-models  described  in  Table  6. 
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Table  6.  This  table  demonstrates  the  meta-model  groupings  considered  in 
determining  optimal  values  for  each  loss  adjustment  factor.  The 
model  naming  convention  used  is  base  year-MAPE-projection  year, 
so  the  model  2007MAPE1  is  the  MAPE  model  for  the  first  projection 
year  (FY07)  of  the  scenario  with  base  year  2007.  The  MAPE  values 
in  this  table  are  for  demonstration  purposes  only. 


grouped  according  to  base  year 

grouped  according  to  projection 
year  being  forecast 
(independent  of  base  year) 

grouped  according  to 
fiscal  year  being  forecast 

2007  scenario  summary 

MAPE  1  summary 

MAPE2007  summary 

model 

MAPE 

model 

MAPE 

model 

MAPE 

2007MAPE  1 

0.12369 

2007MAPE  1 

0.12369 

2007MAPE  1 

0.12369 

2007MAPE  2 

0.16065 

2008MAPE  1 

0.08931 

max 

0.12369 

2007MAPE  3 

0.02441 

2009MAPE  1 

0.12875 

MAPE2008  summary 

2007MAPE  4 

0.00708 

2010MAPE  1 

0.10380 

model 

2007MAPE  5 

0.11844 

2011  MAPE  1 

0.16239 

2007MAPE  2 

0.16065 

max 

0.16065 

max 

0.16239 

2008MAPE  1 

0.08931 

2008  scenario  summary 

MAPE  2  summary 

max 

0.16065 

model 

model 

MAPE  2009  summary 

2008MAPE  1 

0.08931 

2007MAPE  2 

0.16065 

model 

2008MAPE  2 

0.05747 

2008MAPE 2 

0.05747 

2007MAPE  3 

0.02441 

2008MAPE  3 

0.06707 

2009MAPE 2 

0.09360 

2008MAPE  2 

0.05747 

2008MAPE  4 

0.18762 

2010MAPE  2 

0.06817 

2009MAPE  1 

0.12875 

max 

0.18762 

max 

0.16065 

max 

0.12875 

2009  scenario  summary 

MAPE  3  summary 

MAPE  2010  summary 

model 

model 

model 

2009MAPE  1 

0.12875 

2007MAPE  3 

0.02441 

2007MAPE  4 

0.00708 

2009MAPE  2 

0.09360 

2008MAPE 3 

0.06707 

2008MAPE  3 

0.06707 

2009MAPE  3 

0.13666 

2009MAPE 3 

0.13666 

2009MAPE  2 

0.09360 

max 

0.13666 

max 

0.13666 

2010MAPE  1 

0.10380 

2010  scenario  summary 

MAPE  4  summary 

max 

0.10380 

model 

model 

MAPE  2011  summary 

2010MAPE  1 

0.10380 

2007MAPE  4 

0.00708 

model 

2010MAPE 2 

0.06817 

2008MAPE 4 

0.18762 

2007MAPE  5 

0.11844 

max 

0.10380 

max 

0.18762 

2008MAPE  4 

0.18762 

2011  scenario  summary 

MAPE  5  summary 

2009MAPE  3 

0.13666 

model 

model 

2010MAPE  2 

0.06817 

2011  MAPE  1 

0.16239 

2007MAPE  5 

0.11844 

2011  MAPE  1 

0.16239 

max 

0.16239 

max 

0.11844 

max 

0.16239 
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A  solution  for  each  of  the  meta-models  in  Table  6  determines  the  set  of 
loss  adjustment  factors  that  minimizes  the  maximum  MAPE  in  the  meta-model. 
An  overall  meta-model  including  all  15  MAPE  models  also  produces  a  solution 
set.  These  16  solution  sets  demonstrate  a  range  of  values  for  each  loss 
adjustment  parameter.  As  discussed  earlier,  some  parameters  do  not  appear  in 
every  meta-model,  but  this  process  guarantees  at  least  four  data  points  for  every 
loss  adjustment  factor.  Only  one  MAPE  model,  2007MAPE5,  depends  on  the 
values  of  parameters  V73  -  V90.  The  optimization  of  the  three  meta-models  in 
Table  6  including  2007MAPE5,  as  well  as  the  overall  meta-model,  provide  a 
solution  set  for  these  18  factors,  one  data  point  from  each  solution  set.  Similarly, 
parameters  V55  -  V72  appear  only  in  models  2007MAPE4,  2007MAPE5,  and 
2008MAPE4,  so  these  variables  appear  in  seven  solution  sets,  and  their  final 
recommended  values  depend  on  the  seven  data  points  available  for  each 
parameter  in  this  range.  These  data  points  determine  an  average  value  and  95% 
confidence  interval  (Cl)  for  each  loss  adjustment  factor.  The  Cl  width  for  every 
factor  informs  the  decision  to  hold  it  constant  in  subsequent  experiments  or  retain 
it  as  an  input  parameter. 

The  processes  described  in  this  chapter  apply  to  analysis  of  data  collected 
from  each  DOE,  and  the  results  from  the  first  two  DOEs  used  to  formulate  the 
subsequent  DOE.  This  iterative  data  farming  loop  produces  results  that  answer 
the  research  questions  put  forth  in  this  thesis.  Chapter  IV  presents  a  summary  of 
these  results. 
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IV.  ANALYSIS  OF  EXPERIMENTAL  RESULTS 


This  chapter  presents  the  computational  results  derived  from  each  design 
of  experiments  and  discusses  the  process  of  deriving  subsequent  experimental 
designs.  This  chapter  also  presents  results  of  the  final  experiments,  discusses 
the  implications  of  these  results,  and  assesses  the  statistical  and  practical 
significance  of  findings. 

A.  FIRST  DESIGN  OF  EXPERIMENTS 

1.  Loss  Adjustment  Factors  to  Hold  Constant 

A  solution  for  each  of  the  meta-models  in  Table  6  determines  the  set  of 
loss  adjustment  factors  that  minimizes  the  maximum  MAPE  in  the  meta-model. 
An  overall  meta-model  including  all  15  MAPE  models  also  produces  a  solution 
set.  These  16  solution  sets  demonstrate  a  range  of  values  for  each  loss 
adjustment  parameter.  As  discussed  earlier,  some  parameters  do  not  appear  in 
every  meta-model,  but  this  process  guarantees  at  least  four  data  points  for  every 
loss  adjustment  factor. 

This  analysis  calculates  an  average  value  and  95%  confidence  interval 
(Cl)  for  each  loss  adjustment  factor.  The  factors  for  projection  year  five  have  the 
fewest  number  of  data  points,  and  thus  have  the  widest  confidence  intervals,  as 
a  rule.  A  summary  table  of  average  values,  the  Cl  lower  bound  (LB),  and  Cl 
upper  bound  (UB)  for  each  factor  indicates  which  parameters  are  constant  in  the 
next  DOE.  A  rule  of  thumb  used  is  to  hold  constant  factors  with  a  Cl  smaller  than 
0.1,  or  special  cases  of  factors  that  do  not  vary  much,  despite  having  a  large  Cl. 
Table  7  includes  factors  determined  by  DOE1  to  be  constant  in  DOE2.  The 
factors  and  confidence  intervals  in  Table  7  are  final  determined  values,  and 
appear  again  in  the  graphical  summary  of  results  later  in  this  chapter. 
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Table  7.  Factors  to  hold  constant  in  DOE2,  based  on  DOE1  analysis.  Shaded 
cells  have  a  Cl  wider  than  0.1 .  V38  is  held  constant  for  DOE2 
because  9  of  1 1  values  for  V38  were  equal,  with  outliers  heavily 
influencing  Cl  calculation. 


Projection 

Year 

designator 

grade 

input 

parameter 

average 

95%  Confidence  interval 

LB 

UB 

2 

HR 

6 

V24 

1.039 

1.024 

1.054 

902x 

2 

V32 

1.000 

0.999 

1.001 

3 

HR 

2 

V38 

1.208 

1.058 

1.359 

swo 

5 

V47 

1.000 

0.999 

1.001 

902x 

3 

V51 

1.000 

1.000 

1.000 

4 

HR 

4 

V58 

1.000 

1.000 

1.000 

5 

V59 

1.000 

1.000 

1.000 

6 

V60 

1.000 

1.000 

1.000 

swo 

1 

V61 

1.000 

1.000 

1.000 

6 

V66 

1.000 

1.000 

1.000 

902x 

2 

V68 

1.000 

1.000 

1.000 

3 

V69 

1.000 

1.000 

1.000 

5 

HR 

1 

V73 

1.000 

1.000 

1.000 

2 

V74 

1.000 

1.000 

1.000 

5 

V77 

1.000 

1.000 

1.000 

6 

V78 

1.000 

1.000 

1.000 

swo 

2 

V80 

1.000 

1.000 

1.000 

902x 

2 

V86 

1.000 

1.000 

1.000 

3 

V87 

1.000 

1.000 

1.000 

4 

V88 

1.000 

1.000 

1.000 

2.  Selection  of  Parameter  Ranges  for  DOE2 

The  factors  not  included  in  Table  7  remain  variable  input  parameters  in  the 
second  design  of  experiments  (DOE2).  After  analysis  of  DOE1,  some  factors 
have  a  95%  Cl  greater  than  the  size  of  the  original  range  allowed  in  the 
experiment.  The  two-fold  purpose  of  DOE1  is  to  identify  variables  to  hold 
constant  in  subsequent  experiments,  and  to  reduce  the  range  of  remaining  input 
parameters  to  vary  in  DOE2.  With  this  in  mind,  the  range  allowed  for  each 
remaining  variable  is  a  maximum  of  0.2.  The  center  of  each  variable’s  range  is  its 
average  value  determined  from  DOE1  analysis,  with  a  range  of  0.2  or  its  95%  Cl, 
whichever  is  smaller. 
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B.  SECOND  DESIGN  OF  EXPERIMENTS 


1.  Loss  Adjustment  Factors  to  Hold  Constant 

As  in  analysis  of  D0E1,  the  results  of  DOE2  yield  a  solution  for  each  of 
the  meta-models  in  Table  6,  and  an  overall  meta-model  including  all  15  MAPE 
models.  Taken  together,  these  solution  sets  determine  the  set  of  loss  adjustment 
factors  that  minimizes  the  maximum  MAPE  in  the  meta-model.  Descriptive 
statistics  for  these  16  solution  sets  determine  an  average  value  and  a  95%  Cl  for 
each  loss  adjustment  factor.  A  summary  table  of  values  and  CIs  for  each  factor 
indicates  which  factors  are  constant  in  the  next  DOE.  A  rule  of  thumb  used  is  to 
hold  constant  factors  with  a  Cl  smaller  than  0.1.  Table  8  displays  all  loss 
adjustment  factors  to  be  held  constant  in  DOE3,  in  addition  to  those  indentified  in 
Table  7.  The  shaded  rows  in  Table  8  indicate  parameters  held  constant  in  DOE3, 
despite  a  Cl  wider  than  0.1 .  Each  of  these  parameters  is  a  902x  loss  adjustment 
factor,  held  constant  in  DOE3  to  test  the  secondary  hypothesis  of  this  thesis: 
using  distinct  values  for  902x  loss  adjustment  factors  determined  from  a  rigorous 
analysis  yields  a  more  accurate  forecast  than  applying  SWO  loss  adjustment 
factors  to  902x  officers.  The  factors  and  confidence  intervals  in  Table  8  are  final 
determined  values,  and  appear  again  in  the  graphical  summary  of  results  later  in 
this  chapter. 

2.  Selection  of  Parameter  Ranges  for  DOE3 

The  factors  not  included  in  Table  7  or  Table  8  remain  variable  input 
parameters  in  the  third  design  of  experiments  (DOE3).  The  analysis  of  DOE2 
identifies  variables  to  hold  constant  in  subsequent  experiments,  and  to  reduce 
the  range  of  remaining  input  parameters  to  vary  in  DOE3.  With  this  in  mind,  all 
SWO  and  HR  loss  adjustment  factors  with  a  Cl  wider  than  0.1  remain  variable  in 
DOE3.  All  902x  loss  adjustment  factors  are  constant  in  DOE3,  to  test  the 
secondary  hypothesis  of  this  thesis:  using  distinct  values  for  902x  loss 
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adjustment  factors  determined  from  a  rigorous  analysis  yields  a  more  accurate 
forecast  than  applying  SWO  loss  adjustment  factors  to  902x  officers. 


Table  8.  Factors  to  hold  constant  in  DOE3,  based  on  analysis  of  DOE2. 

Shaded  cells  have  Cl  wider  than  0.1,  but  remain  constant  in  the  next 
DOE  at  their  average  values,  for  the  sole  purpose  of  testing  the 
secondary  hypothesis:  that  experimentally  determined  factors  for 
902x  will  yield  a  better  forecast  than  using  the  SWO  factors  for  902x. 


Projection 

designator 

grade 

input 

average 

95%  Confidence  interval 

Year 

parameter 

LB 

UB 

HR 

1 

VI 

1.001 

0.985 

1.018 

2 

V2 

1.004 

0.972 

1.035 

2 

V14 

0.970 

0.812 

1.127 

1 

3 

V15 

1.051 

0.882 

1.220 

902x 

4 

V16 

1.012 

0.945 

1.079 

5 

V17 

1.000 

0.964 

1.035 

6 

V18 

0.996 

0.937 

1.055 

HR 

2 

V20 

1.067 

1.051 

1.083 

3 

V21 

1.087 

1.074 

1.099 

SWO 

3 

V27 

0.883 

0.857 

0.909 

2 

3 

V33 

1.015 

0.980 

1.049 

902x 

4 

V34 

1.019 

0.963 

1.075 

5 

V35 

1.036 

0.991 

1.082 

6 

V36 

1.016 

1.013 

1.019 

1 

V37 

1.078 

1.078 

1.078 

HR 

4 

V40 

1.046 

1.030 

1.061 

5 

V41 

1.063 

1.059 

1.067 

6 

V42 

1.077 

1.067 

1.086 

1 

V43 

1.045 

1.016 

1.075 

3 

SWO 

4 

V46 

1.047 

1.011 

1.084 

6 

V48 

1.072 

1.072 

1.072 

2 

V50 

1.078 

1.051 

1.105 

902x 

4 

V52 

1.079 

1.067 

1.090 

5 

V53 

1.016 

0.982 

1.050 

6 

V54 

1.014 

0.979 

1.049 

1 

V55 

1.075 

1.075 

1.075 

HR 

2 

V56 

1.118 

1.100 

1.135 

3 

V57 

1.149 

1.149 

1.149 

4 

SWO 

4 

V64 

1.101 

1.098 

1.104 

4 

V70 

1.110 

1.110 

1.110 

902x 

5 

V71 

1.097 

1.075 

1.120 

6 

V72 

1.141 

1.056 

1.225 

HR 

4 

V76 

1.116 

1.116 

1.116 

SWO 

1 

V79 

1.183 

1.183 

1.183 

5 

6 

V84 

1.209 

1.209 

1.209 

902x 

5 

V89 

1.094 

1.094 

1.094 

6 

V90 

1.119 

1.119 

1.119 
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The  remaining  28  SWO  and  HR  loss  adjustment  factors  vary  in  DOE3 
according  to  an  NOLH  design.  The  128  runs  of  this  design  execute  twice;  DOE3a 
assigns  to  all  902x  loss  adjustment  factors  the  value  determined  in  analysis  of 
DOE1  and  DOE2,  while  DOE3b  sets  each  902x  loss  adjustment  factor  equal  to 
the  SWO  loss  adjustment  factor  for  the  same  projection  year  and  grade 
combination. 

C.  THIRD  DESIGN  OF  EXPERIMENTS 

1.  Use  of  Unique  Loss  Adjustment  Factors  for  Declined  Lateral 
Transfer  Applicants 

The  null  hypothesis  for  this  comparison  is  the  secondary  hypothesis 
specified  in  Chapter  I  of  this  thesis:  that  using  distinct  values  for  902x  loss 
adjustment  factors  determined  from  a  rigorous  analysis  yields  a  more  accurate 
forecast  than  applying  SWO  loss  adjustment  factors  to  902x  officers.  A  paired 
two  sample  t-test  is  sufficient  to  test  this  hypothesis.  The  results  of  this  t-test, 
displayed  in  Table  9,  indicate  that  using  experimentally  determined  902x  loss 
adjustment  factors  does  not  provide  a  more  accurate  forecast,  as  measured  by 
MAPE,  than  setting  all  902x  factors  equal  to  their  corresponding  SWO  factors,  at 
a  95%  confidence  level. 


Table  9.  t-Test:  Paired  Two  Sample  for  Means,  to  test  secondary  hypothesis: 

using  distinct  values  for  902x  loss  adjustment  factors  determined 
from  a  rigorous  analysis  yields  a  more  accurate  forecast  (smaller 
MAPE)  than  applying  SWO  loss  adjustment  factors  to  902x  officers. 


DOE3a 

DOE3b 

Mean 

0.1038115 

0.09837249 

Variance 

0.00071632 

0.00057733 

Observations 

16 

16 

Pearson  Correlation 

0.93988152 

Hypothesized  Mean 

0 

df 

15 

t  Stat 

2.50570874 

P(T<=t)  one-tail 

0.01134058 

t  Critical  one-tail 

1.73960672 

39 


This  important  result  indicates  there  is  no  benefit  to  implementing  a 
distinct  set  of  loss  adjustment  factors  for  SWOs  declined  a  lateral  transfer. 
Notably,  OSAM  models  historical  loss  rates  differently  for  902x  officers  than  for 
SWOs  who  have  not  applied  for  a  lateral  transfer.  Thus,  disproval  of  this 
secondary  research  hypothesis  does  not  mean  there  is  no  benefit  to  modeling 
902x  officers  distinctly  from  SWOs.  Additional  model  exploration  may  offer  a 
conclusive  answer  to  this  question. 

2.  Final  Values  for  Loss  Adjustment  Factors 

As  in  earlier  phases  of  analysis,  a  solution  for  each  of  the  meta-models  in 
Table  6,  and  an  overall  meta-model  including  all  15  MAPE  models,  determines 
the  set  of  loss  adjustment  factors  that  minimizes  the  maximum  MAPE  in  the 
meta-model.  Subsequent  to  disproval  of  the  secondary  research  hypothesis, 
analysis  DOE3b  recommends  appropriate  factor  values  for  future  forecasting, 
disregarding  results  from  DOE3a.  Descriptive  statistics  for  the  16  solution  sets  of 
DOE3b  determine  an  average  value  and  a  95%  Cl  for  each  loss  adjustment 
factor.  A  summary  table  of  values  and  CIs  for  each  factor  indicates  final 
recommended  values  of  loss  adjustment  factors,  and  their  respective  confidence 
intervals  (See  Table  10).  The  shaded  rows  in  Table  10  indicate  parameters  with 
a  Cl  wider  than  0.15.  Additional  experiments  might  narrow  these  ranges  further, 
but  the  reduction  in  parameters  varied  through  this  iterative  analytical  process  led 
to  difficulty  generating  acceptable  regression  models.  This  research  instead 
recommends  accepting  these  average  values,  with  a  note  of  caution.  The  factors 
and  confidence  intervals  in  Table  8  are  final  determined  values,  and  appear 
again  in  the  graphical  summary  of  results  later  in  this  chapter. 

Taken  together,  the  values  in  Tables  7-8  and  Table  10  are  the 
experimental  results  determined  for  use  in  future  forecasting  efforts  with  OSAM. 
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Table  1 0.  Factor  values  determined  by  analysis  of  third  and  final  DOE.  Shaded 
cells  have  Cl  wider  than  0.15. 


Projection 

designator 

grade 

input 

average 

95%  Confidence  interval 

Year 

parameter 

LB 

UB 

3 

V3 

0.981 

0.938 

1.024 

HR 

4 

V4 

0.984 

0.919 

1.050 

5 

V5 

1.031 

0.961 

1.101 

6 

V6 

1.020 

0.976 

1.064 

1 

1 

V7 

1.028 

0.908 

1.147 

2 

V8 

0.997 

0.897 

1.096 

SWO 

3 

V9 

0.974 

0.905 

1.043 

4 

V10 

0.999 

0.903 

1.096 

5 

Vll 

0.987 

0.903 

1.072 

6 

V12 

0.961 

0.828 

1.095 

1 

V19 

1.020 

1.018 

1.022 

HR 

4 

V22 

0.999 

0.960 

1.038 

5 

V23 

0.983 

0.957 

1.010 

2 

1 

V25 

0.991 

0.955 

1.028 

2 

V26 

1.000 

0.942 

1.057 

SWO 

4 

V28 

1.008 

0.972 

1.043 

5 

V29 

1.004 

0.931 

1.077 

6 

V30 

1.014 

0.966 

1.061 

HR 

3 

V39 

1.092 

1.005 

1.179 

3 

SWO 

2 

V44 

1.042 

1.042 

1.042 

3 

V45 

0.874 

0.794 

0.954 

2 

V62 

0.961 

0.912 

1.009 

4 

SWO 

3 

V63 

0.936 

0.831 

1.042 

5 

V65 

0.992 

0.963 

1.021 

HR 

3 

V75 

1.101 

1.082 

1.119 

5 

3 

V81 

0.902 

0.834 

0.969 

SWO 

4 

V82 

1.017 

0.981 

1.054 

5 

V83 

1.173 

1.173 

1.173 

3.  Interpretation  of  Results 

The  final  experimental  values  displayed  in  Tables  7-8  and  Table  10  are 
useful  for  implementation  in  OSAM,  but  a  list  of  numbers  is  only  one 
demonstration  of  how  the  recommendations  of  this  research  compare  to  the 
values  used  in  OSAM  runs  prior  to  this  analysis.  Figure  2  is  a  visual 
representation  of  the  95%  CIs  for  SWO  loss  adjustment  factors,  overlaid  with  the 
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null  hypothesis  (primary  hypothesis  1)  and  the  working  hypothesis  (primary 
hypothesis  2).  The  vertically  aligned  diamonds  on  this  plot  each  represent  the 
upper  and  lower  bound  of  the  Cl  for  the  case  indicates  on  the  horizontal  axis. 
Figure  2  shows  that  the  Cl  for  many  SWO  loss  adjustment  factors  contains  either 
one  or  both  hypothetical  values,  with  some  notable  exceptions. 

The  03,  or  Lieutenant,  average  loss  adjustment  factors  recommended  by 
this  analysis  fall  below  the  null  and  working  hypothesis  for  all  projection  years, 
with  the  working  hypothesis  values  falling  completely  outside  the  Cl  for  projection 
years  two  through  five,  and  the  null  hypothesis  values  falling  completely  outside 
the  Cl  for  projection  years  two,  three,  and  five.  This  result  could  be  due  to  a 
tendency  of  OSAM  to  overestimate  03  loss  rates,  leading  to  a  recommendation 
for  03  loss  adjustment  factors  less  than  1.0  across  all  projection  years.  This 
finding  may  suggest  a  need  to  revisit  the  method  of  estimating  03  loss  rates  in 
OSAM,  but  more  likely  identifies  a  specific  issue  with  SWOs,  because  the  same 
behavior  does  not  occur  in  the  HR  community  (See  Figure  3). 


95%  Confidence  Interval  of  optimal  loss  adjustment  factors  for 
SWOs  compared  to  null  and  working  hypotheses 
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Figure  2.  Ninety-five  percent  confidence  intervals  of  optimal  loss  adjustment 
factors  for  SWOs  compared  to  null  and  working  hypotheses. 
Diamonds  indicate  upper  and  lower  bound  of  respective  CIs. 
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Figure  3  is  a  visual  representation  of  the  95%  CIs  for  HR  loss  adjustment 
factors,  overlaid  with  the  null  hypothesis  (primary  hypothesis  1 )  and  the  working 
hypothesis  (primary  hypothesis  2).  The  vertically  aligned  diamonds  on  this  plot 
each  represent  the  upper  and  lower  bound  of  the  Cl  for  the  case  indicates  on  the 
horizontal  axis.  Figure  3  shows  that  the  Cl  for  many  HR  loss  adjustment  factors 
contains  either  the  null  or  working  hypothesis  value.  The  HR  factor  values  align 
closely  with  the  working  hypothesis  values,  with  the  exception  of  several  values 
in  projection  years  four  and  five  eliminated  from  further  variation  early  in  analysis 
because  they  did  not  appear  in  any  regression  models. 

Figure  3  suggests  that  OSAM  may  be  overestimating  loss  rates  for  HR 
02s  and  03s  in  projection  years  two  and  four,  and  underestimating  loss  rates  for 
HR  01s,  04s,  and  05s  in  projection  year  two.  While  the  deviations  from 
hypothetical  values  are  not  as  large  as  for  SWOs,  there  is  evidence  to  suggest 
forecast  accuracy  improves  when  using  the  experimentally  determined  loss 
adjustment  factors,  particularly  in  years  two  and  four. 


Figure  3.  Ninety-five  percent  confidence  intervals  of  optimal  loss  adjustment 

factors  for  HRs  compared  to  null  and  working  hypotheses.  Diamonds 
indicate  upper  and  lower  bound  of  respective  CIs. 
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The  number  of  HR  officers  is  significantly  smaller  than  the  number  of 
SWOs  at  any  given  time.  This  analysis  determines  the  loss  adjustments  factors 
displayed  in  Figures  2  and  3  simultaneously.  The  focus  of  this  thesis  is  overall 
MAPE  for  the  conglomerate  of  officer  communities  considered,  but  determination 
of  loss  adjustment  factors  one  designator  at  a  time  could  result  in  detailed 
information  for  specific  communities.  Development  of  regression  models  for 
MAPE  separated  by  designator  will  yield  this  information. 

D.  VERIFICATION  EXPERIMENT 

A  final  experiment  runs  with  only  three  design  points.  The  first  DP  sets  all 
loss  adjustment  factors  of  interest  equal  to  1.0,  to  test  primary  hypothesis  1, 
referred  to  in  Figures  2  and  3  as  the  null  hypothesis.  The  second  DP  sets  loss 
adjustment  factors  for  projection  year  one  equal  to  1.0,  1.041  in  year  two,  1.059 
in  year  three,  1 .085  in  year  four,  and  1.114  in  year  five,  to  test  primary  hypothesis 
2,  referred  to  in  Figures  2  and  3  as  the  working  hypothesis.  The  third  DP  sets  all 
loss  adjustment  factors  equal  to  the  average  values  reported  in  Tables  7-8  and 
Table  10.  Each  of  these  DPs  provides  15  MAPE  values,  one  for  each  base 
year/projection  year  combination.  These  three  sets  of  MAPE  values,  shown  in 
Table  1 1 ,  are  comparable  to  one  another  for  hypothesis  testing. 
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Table  1 1 . 


Summary  of  MAPE  values  from  final  set  of  OSAM  runs. 


MAPE 

baseyear 

projection 

year 

hypothesis 

1 

hypothesis 

2 

Experimental 

Results 

2007 

2007 

0.03994 

0.03994 

0.04063 

2007 

2008 

0.05722 

0.05793 

0.0649 

2007 

2009 

0.08593 

0.0876 

0.09325 

2007 

2010 

0.1308 

0.13008 

0.11542 

2007 

2011 

0.16801 

0.17947 

0.15208 

2008 

2008 

0.05485 

0.05485 

0.05305 

2008 

2009 

0.07957 

0.07965 

0.08988 

2008 

2010 

0.10263 

0.1096 

0.09576 

2008 

2011 

0.16301 

0.16365 

0.14534 

2009 

2009 

0.07982 

0.07982 

0.07822 

2009 

2010 

0.11527 

0.11616 

0.114 

2009 

2011 

0.16658 

0.16481 

0.15079 

2010 

2010 

0.09616 

0.09616 

0.09363 

2010 

2011 

0.12217 

0.12119 

0.12048 

2011 

2011 

0.10249 

0.10249 

0.10633 

1.  Test  of  Hypothesis  1 

The  null  hypothesis  for  this  comparison  is  that  the  MAPE  generated  from 
experimental  results  is  not  smaller  than  the  MAPE  generated  from  a  set  of  loss 
adjustment  factors  in  which  all  values  are  1.0.  A  paired  two  sample  t-test  is 
sufficient  to  test  this  hypothesis.  The  results  of  this  t-test,  displayed  in  Table  12, 
indicate  that  experimental  results  provide  a  better  forecast  than  setting  all  loss 
adjustment  factors  equal  to  1 .0,  at  a  90%  confidence  level. 
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Table  12.  t-Test:  Paired  Two  Sample  for  Means,  to  test  primary  hypothesis  1 : 

the  MAPE  generated  from  experimental  results  is  not  smaller  than 
the  MAPE  generated  from  a  set  of  loss  adjustment  factors  in  which 
all  values  are  1 .0. 


Experimental 

Results 

hypothesis  1 

Mean 

0.100917333 

0.104296667 

Va  riance 

0.001133595 

0.001639373 

Observations 

15 

15 

Pearson  Correlation 

0.986185896 

Hypothesized  Mean  Difference 

0 

degrees  of  freedom 

14 

t  Stat 

-1.426505455 

P(T<=t)  one-tail 

0.087820762 

t  Critical  one-tail 

1.761310115 

2.  Test  of  Hypothesis  2 

The  null  hypothesis  for  this  comparison  is  that  the  MAPE  generated  from 
experimental  results  is  not  smaller  than  the  MAPE  generated  from  a  set  of  loss 
adjustment  factors  in  which  values  for  projection  year  one  are  equal  to  1.0,  year 
two  values  are  1.041,  year  three  values  are  1.059,  year  four  values  are  1.085, 
and  year  five  values  are  1.114.  Again,  a  paired  two  sample  t-test  is  sufficient  to 
test  this  hypothesis.  The  results  of  this  t-test,  displayed  in  Table  13,  indicate  that 
experimental  results  provide  a  better  forecast  than  applying  OPNAV  N14’s  loss 
adjustment  factors  describing  a  slowly  improving  economy  (setting  values  for 
projection  year  one  equal  to  1 .0,  year  two  values  are  1 .041 ,  year  three  values  are 
1.059,  year  four  values  are  1.085,  and  year  five  values  are  1.114.),  at  a  90% 
confidence  level. 
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Table  13. 


t-Test:  Paired  Two  Sample  for  Means,  to  test  primary  hypothesis  2: 
the  MAPE  generated  from  experimental  results  is  not  smaller  than 
the  MAPE  generated  from  a  set  of  loss  adjustment  factors  describing 
a  slowly  improving  economy. 


Experimental 

Results 

hypothesis  2 

Mean 

0.100917333 

0.10556 

Va  riance 

0.001133595 

0.00173016 

Observations 

15 

15 

Pearson  Correlation 

0.982063123 

Hypothesized  Mean  Difference 

0 

degrees  of  freedom 

14 

t  Stat 

-1.691014156 

P(T<=t)  one-tail 

0.056482551 

t  Critical  one-tail 

1.761310115 

The  designs  of  experiments  used  in  this  thesis  target  analysis  of  data  from 
base  years  2007  through  2011,  and  this  thesis  has  proven  that  changing  loss 
adjustment  factor  values  can  generate  a  better  forecast  (measured  by  MAPE) 
than  applying  the  same  values  to  all  loss  adjustment  factors.  It  is  desirable  to  test 
these  results  against  forecasts  not  included  in  the  original  analysis,  but  at  this 
time  additional  data  is  unavailable.  Nonetheless,  this  research  demonstrates  the 
benefits  of  applying  data  farming  to  OSAM,  both  to  validate  the  model  and 
improve  it  for  future  use.  The  fifth  and  final  chapter  of  this  thesis  delineates 
significant  findings  and  suggestions  for  future  research. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


This  thesis  demonstrates  the  potential  to  assess  and  improve  OSAM  with 
insights  provided  through  data  farming.  This  chapter  summarizes  specific 
findings  related  to  SWO  and  HR  loss  adjustment  factors,  and  suggests  direction 
for  further  exploration  of  OSAM’s  input  parameters.  OSAM  utilizes  input 
parameters  contained  in  60  database  files;  while  this  research  varied  a  relatively 
small  number  of  parameters  in  only  one  of  those  files,  the  methods  employed  to 
measure  accuracy,  generate  a  better  forecast,  and  test  specific  research 
questions  lay  the  foundation  for  countless  further  applications  to  OSAM  and  to 
other  models  owned  by  OPNAV  N14.  This  thesis  takes  necessary  first  steps  to 
explore  the  applicability  of  data  farming,  and  the  results  suggest  numerous 
applications  for  future  research  to  undertake. 

A.  SIGNIFICANT  FINDINGS 

1.  Assessment  of  Loss  Rate  Variation 

OSAM  implements  loss  rate  variation  over  a  forecast  period  through  the 
multiplication  of  historical  loss  rates  by  a  loss  adjustment  factor.  There  is  a 
potential  to  model  unique  loss  adjustment  factors  for  each  combination  of 
designator,  projection  year,  and  years  of  commissioned  service  (YCS).  The 
rigorous  numerical  analysis  in  this  thesis  comparing  simulation  forecasts  to 
historical  inventories  confirms  that  in  many  cases,  officer  loss  rates  next  year  will 
be  similar  to  loss  rates  this  year.  The  average  experimentally  determined  SWO 
and  HR  loss  adjustment  factors  for  the  first  projection  year  range  between  0.961 
and  1.031,  and  the  95%  confidence  interval  includes  the  value  1.0  for  every 
analyzed  loss  adjustment  factor.  This  finding  is  in  keeping  with  common  practice 
in  running  OSAM  simulations. 

Forecast  analysis  in  projection  years  two  through  five  yields  less 
consistent  results  across  the  set  of  HR  and  SWO  loss  adjustment  factors  varied. 

49 


Recommended  SWO  Lieutenant  (03)  loss  adjustment  factors  are  less  than  1 .0  in 
every  projection  year,  and  multiple  recommended  values  are  very  close  to  1 .0 
even  in  later  projection  years.  For  loss  adjustment  factors  that  deviate  from  1 .0, 
there  seems  to  be  a  general  upward  trend  with  increasing  projection  years. 

2.  Validation  of  Results 

This  research  delivers  a  suggested  value  to  apply  for  each  factor  in  future 
OSAM  runs,  and  provides  a  Cl  for  these  factors.  The  complex  interactions 
between  officer  communities,  ranks,  and  overall  inventory  make  validation  of 
individual  loss  adjustment  factors  impractical.  A  validation  of  the  entire  solution 
set  is  feasible,  and  demonstrates  that  experimentally  determined  loss  adjustment 
factors  yield  a  better  forecast  than  holding  parameters  constant  equal  to  1 .0,  at  a 
90%  confidence  level.  This  validation  also  affirms  that  experimental  results  yield 
a  better  forecast  than  varying  all  factors  identically  in  each  projection  year 
according  to  OPNAV  N14’s  set  of  loss  adjustment  factors  describing  a  slowly 
improving  economy  (1.0  in  year  one,  1.041  in  year  two,  1.059  in  year  three, 
1 .085  in  year  four,  and  1 .1 14  in  year  five),  again  at  a  90%  confidence  level. 

3.  Loss  Rate  Variation  for  Declined  Lateral  Transfer  Applicants 

As  part  of  this  research,  a  modification  of  OSAM  adds  a  fictitious 
designator,  902x.  This  designator  is  adaptable  to  pursue  various  research 
questions,  and  for  this  thesis  represents  SWOs  declined  the  opportunity  to  lateral 
transfer.  Analysis  of  the  first  two  experimental  designs  determines  optimal  values 
for  902x  loss  adjustment  factors,  and  the  final  design  of  experiments  executed 
twice,  once  with  these  experimentally  determined  values,  and  once  with  all  902x 
factors  equal  to  their  corresponding  SWO  factors.  Analysis  of  this  experiment 
reveals  that  there  is  no  need  to  model  loss  adjustment  factors  differently  for 
declined  lateral  transfer  applicants.  In  fact,  using  the  experimentally  determined 
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902x  values  does  not  provide  a  more  accurate  forecast,  as  measured  by  MAPE, 
than  applying  the  appropriate  SWO  values  to  these  factors,  at  a  95%  confidence 
level. 

This  result  at  first  seems  counterintuitive,  since  earlier  research  (Kleyman 
&  Parcell,  2010)  shows  that  SWOs  declined  the  opportunity  to  lateral  transfer  are 
more  likely  to  leave  the  Navy  than  SWOs  who  never  applied  for  lateral  transfer. 
With  this  background  in  mind,  it  is  important  to  recall  that  this  hypothesis  test 
indicates  there  is  no  benefit  to  varying  loss  adjustment  factors  distinctly  for  902x 
officers,  but  does  not  consider  the  underlying  loss  rates  built  into  the  model. 
Notably,  OSAM  models  902x  officers  with  unique  historical  loss  rates,  so  this 
interesting  finding  is  insufficient  to  suggest  eliminating  the  902x  designator  from 
the  model. 

4.  Applicability  of  Data  Farming  to  OSAM 

One  objective  of  this  thesis  is  to  explore  the  potential  for  continued 
application  of  data  farming  techniques  to  officer  inventory  projection.  OSAM’s 
adaptation  for  use  in  a  data  farming  environment  enabled  multiple  simulations  to 
run  without  operator  interaction.  This  capability  provides  an  opportunity  to  run 
countless  variations  of  experimental  designs  in  future  explorations  of  the  model. 
Data  farming  is  unquestionably  a  useful  technique  for  exploring  OSAM. 

B.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

1.  Additional  Base  Years 

A  noted  limitation  of  this  research  is  the  lack  of  aligning  forecast  and 

historical  data  for  early  fiscal  years  and  later  projection  years  in  the  time  period 

observed.  The  analytical  approach  employed  adjusts  for  this  information 

shortage  by  considering  overlapping  meta-models.  The  analysis  of  forecasts  for 

additional  base  years  is  an  ideal  solution  to  this  data  scarcity.  In  addition  to 

improving  the  accuracy  of  results,  an  expansion  of  this  thesis  to  additional  base 
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years  provides  an  opportunity  to  observe  forecast  variation  over  additional 
economic  and  political  environments,  which  may  influence  the  retention  behavior 
of  some  Navy  officers. 

Analysis  of  forecast  accuracy  over  a  longer  time  period  should  reduce  the 
number  of  meta-models  needed  to  determine  recommended  loss  adjustment 
factors.  Additional  data  for  each  meta-model  should  also  enable  assessment  of 
which  set  of  meta-model  groupings  is  most  effective  in  determining 
recommended  loss  adjustment  factors.  An  exploration  of  meta-models  is 
particularly  useful  to  OPNAV  N14,  as  this  methodology  may  apply  to  future 
versions  of  OSAM  as  a  built-in  accuracy  check. 

2.  Modeling  Distinct  Loss  Rates  for  Declined  Lateral  Transfer 
Applicants 

This  research  concludes  there  is  no  need  to  model  loss  adjustment  factors 
for  declined  lateral  transfer  applicants  uniquely  from  the  loss  adjustment  factors 
of  officers  who  did  not  apply  from  lateral  transfer.  A  logical  next  step  is  to  assess 
whether  there  is  value  in  modeling  these  declined  lateral  transfer  applicants  as  a 
unique  designator  at  all.  As  noted  previously,  OSAM  models  902x  officers  with 
unique  historical  loss  rates.  The  implementation  of  the  902x  designator  in  OSAM 
is  at  the  user’s  discretion,  and  its  definition  is  alterable  according  to  the 
researcher’s  needs.  Running  a  DOE  twice  (once  with  902x  historical  loss  rates 
determined  appropriately  for  SWOs  declined  the  opportunity  to  lateral  transfer, 
and  once  with  902x  historical  loss  rates  equal  to  corresponding  SWO  historical 
loss  rates),  will  provide  the  data  necessary  to  resolve  this  research  question. 
Past  research  suggests  these  officers  have  unique  promotion  and  loss 
probabilities.  Application  of  data  farming  to  this  problem  can  determine  whether 
these  differences  are  significant  enough  to  warrant  modeling  a  unique 
designator.  Application  of  this  methodology  to  additional  officer  communities  may 
determine  which,  if  any,  communities  benefit  from  modeling  this  category  of 
officers  uniquely. 
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3. 


Additional  Designators 


This  research  studies  the  loss  adjustment  factors  for  four  of  the  74 
designators  modeled  by  OSAM.  There  is  value  in  repeating  this  study  for  other 
designators.  The  methodology  applied  in  this  thesis  is  appropriate  to  recommend 
loss  adjustment  factors  for  the  designators  studied,  but  all  officer  communities 
interact  in  OSAM  to  meet  the  total  inventory  requirements  imposed  on  the  entire 
Navy  Officer  Corps.  A  similar  study  encompassing  more  designators  may 
recommend  different  values  for  the  loss  adjustment  factors  studied  in  this  thesis, 
due  to  such  interactions.  Data  farming  methods  are  ideal  for  tackling  this  type  of 
problem  expansion,  enabling  exploration  of  a  complex,  unwieldy  design  space. 

Data  farming  methods  continue  to  evolve,  and  there  are  some  potential 
limitations  imposed  by  computing  power  and  designs  of  experiments  for  large 
numbers  of  input  parameters.  OSAM  includes  2,025  loss  adjustment  factors,  and 
the  variation  of  all  factors  simultaneously  may  present  challenges  in  these  areas. 
Exploration  of  the  design  space  is  possible  without  unique  variation  of  all  2,025 
factors,  with  information  about  which  designators  have  loss  rates  that  vary 
similarly  from  year  to  year.  Grouping  designators  together  by  categories  of 
similar  behavior  could  enable  a  study  of  all  loss  adjustment  factors 
simultaneously.  For  instance,  it  may  be  appropriate  to  assign  the  same  loss 
adjustment  factors  to  all  Limited  Duty  Officers  (LDOs),  a  separate  set  of  loss 
adjustment  factors  to  all  nurses  and  Medical  Service  Corps  (MSC)  officers,  and 
yet  another  set  of  loss  adjustment  factors  to  all  Restricted  Line  (RL)  officers.  A 
numerically  determined  solution  set  for  all  loss  adjustment  factors  in  OSAM  could 
greatly  improve  the  forecast  accuracy  of  mid-term  inventory  projections. 

4.  Model  MAPE  by  Designator 

This  thesis  assesses  the  quality  of  forecasts  by  measuring  mean  absolute 

proportional  error  (MAPE),  proportioned  on  rank.  Application  of  this  measure  of 

effectiveness  to  more  specific  modeling  may  yield  additional  findings  of  interest 

to  both  OPNAV  N14  and  individual  officer  communities.  For  example, 
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proportioning  MAPE  on  both  designator  and  rank  would  enable  the  building  of 
separate  SWO  and  HR  MAPE  models.  Both  sets  of  models  should  still  include 
loss  adjustment  factors  from  both  communities,  as  their  inventories  interact 
during  the  simulation.  The  unique  solution  sets  generated  from  this  additional 
analysis  may  provide  insight  on  the  loss  behavior  of  individual  communities,  and 
can  identify  the  optimal  set  of  loss  adjustment  factors  for  a  goal  of  maximizing 
forecast  accuracy  of  a  single  officer  community.  Similarly,  both  communities’ 
models  could  be  considered  together,  weighting  the  MAPE  models  appropriately 
according  to  the  Navy’s  need  to  meet  one  community’s  inventory  needs  more 
than  another.  This  task  requires  significant  input  from  Navy  personnel  subject 
matter  experts,  or  consideration  of  many  different  weighting  combinations.  This 
approach  to  analysis  has  great  potential  for  informing  policy  decisions,  but  the 
selection  of  appropriate  weights  for  different  officer  communities  is  certain  to  vary 
widely  between  subject  matter  experts. 

5.  Weighting  Forecast  Accuracy 

Accurate  projection  of  Navy  officer  inventory  is  a  goal  of  OSAM,  and  this 
thesis  focuses  on  enhancing  efforts  to  achieve  this  objective.  In  addition  to 
overall  forecast  accuracy,  there  are  many  officer  inventory  goals  that  individual 
communities  strive  to  meet.  For  instance,  the  SWO  community  needs  a  minimum 
number  of  Lieutenants  at  seven  YCS  to  fill  department  head  afloat  billets;  SWO 
accession  plans  intend  to  meet  this  need,  even  though  this  may  result  in  more 
Ensigns  than  needed  in  the  interim  (Monroe  &  Cymrot,  2004).  Data  farming 
OSAM  and  analyzing  MAPE  calculations  for  very  specific  combinations  of  officer 
attributes  can  potentially  improve  projection  to  meet  such  specific  goals. 

C.  IMPACT  OF  RESEARCH 

OPNAV  N14  gleans  multiple  benefits  from  this  research,  both  in  the  short 
term  and  the  long  term.  Analysts  at  N14  have  envisioned  building  an  automated 
validation  or  accuracy  check  capability  into  current  and  future  forecasting 
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models,  and  will  explore  the  use  of  MAPE  as  a  forecast  accuracy  metric.  The 
methodology  utilized  in  this  thesis  seems  a  promising  means  of  accomplishing 
this  goal. 

This  research  highlights  some  weaknesses  of  OSAM  that  N14  can 
improve  upon.  One  recommendation  of  this  study  is  to  include  more  historical 
data  to  adequately  measure  accuracy;  N14  plans  to  make  this  a  requirement  in 
future  model  development.  This  thesis  also  identifies  OSAM’s  overestimation  of 
SWO  Lieutenant  loss  rates  in  all  projection  years.  The  loss  estimation  process  is 
a  key  step  in  personnel  forecasting,  and  N14  strives  to  model  losses  well.  The 
next  update  of  OSAM  will  strive  to  correct  this  weakness,  either  including  more 
explanatory  historic  information  (e.g.,  years  of  total  service),  or  using  new  rate 
generation  techniques  (e.g.,  machine  learning,  Bayesian,  or  agent-based 
behavioral  models). 

To  leverage  the  conclusion  that  forecast  accuracy  can  improve  with 
experimentally  determined  loss  adjustment  factors,  N14  will  ensure  that  future 
models  retain  the  capability  for  users  to  modify  loss  rates.  This  potential  benefit 
lends  itself  to  future  research,  particularly  the  investigation  of  loss  rates  for 
additional  specific  officer  communities. 

This  thesis,  the  first  application  of  data  farming  concepts  to  OSAM,  lays 
the  groundwork  for  continuing  efforts  in  this  area.  The  conclusions  of  this 
research  provide  actionable  steps  for  OPNAV  N14  to  pursue.  Further,  the 
implementation  of  designs  of  experiments,  selection  of  a  forecast  accuracy 
metric,  and  development  of  a  multi-faceted  analytical  approach  employed  in  this 
study  serve  as  a  roadmap  for  future  exploration  of  OSAM. 
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APPENDIX 


This  appendix  describes  the  data  farming  components  used  to  data  farm 
OSAM.  The  information  included  here  is  derived  from  the  notes  of  Stephen 
Upton,  who  developed  the  code  used  and  executed  all  experiments  in  this  thesis 
on  the  computer  clusters  of  the  SEED  Center  for  Data  Farming. 

The  current  implementation  of  data  farming  OSAM  is  a  limited  version,  in 
that  the  development  of  code  and  other  artifacts  is  specifically  to  assist  in 
completion  of  this  thesis.  The  hard  coding  of  some  settings  supports  timely 
implementation  of  OSAM  in  a  data  farming  environment.  In  particular,  the  design 
points  used  in  the  experiments  varied  only  factors  in  the  loss_adj.dbf  file,  based 
on  settings  for  fiscal  year,  grade,  and  designator.  Using  the  current 
implementation,  one  can  vary  a  factor  for  any  combination  of  fiscal  year,  grade, 
or  designator  contained  in  the  loss_adj.dbf  file.  Extension  of  data  farming  to 
varying  factors  in  any  other  input  file  will  require  editing  the  implementation  code. 

OSAM  simulations  run  via  a  Microsoft  Visual  Fox  Pro  Version  9  (VFP9) 
executable  file.  An  additional  VFP9  file,  dfosam.exe,  assists  in  data  farming 
OSAM.  A  licensed  copy  of  VFP9  is  necessary  to  edit  the  OSAM  input  files,  as 
indicated  by  designs  of  experiments.  In  addition  to  this  new  VFP9  executable  file, 
several  other  elements  are  part  of  the  vital  infrastructure  for  data  farming  OSAM. 
Table  14  provides  a  summary  of  these  components. 

Each  OSAM  run  saves  the  output  files,  Multi-Year  Summary. dbf  and 
flow_pt.dbf,  in  a  directory  named  for  the  appropriate  design  point.  These  are  the 
files  containing  data  to  be  processed  and  analyzes  as  described  in  Chapter  III. 
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Table  14. 


Summary  of  components  utilized  in  the  implementation  of  OSAM  in  a 
data  farming  environment. 


Component 

Description 

stage. OSAM. run. R 

this  code  creates  the  Output  and  submit  directories,  then 
creates  a  DP_X  directory  for  all  DPs  in  the  design,  where 
X  is  the  design  point  or  row  number  of  the  DOE  file.  It 
then  creates  condor_submit  files  for  all  the  DPs  and  puts 
them  in  the  submit  directory,  (this  code  needs  4 
arguments:  [1]  study. dir  is  the  directory  location,  [2] 
basecase.name  is  the  zipped  contents  of  an  OSAM 
scenario,  including  the  OSAM  executable  file,  [3] 
start. year,  [4]  doe. file. name) 

submit-template.data 

this  is  the  condor  submit  template,  with  BASECASE 
STARTYEAR  DOEFILE  DPNUM  and  STUDY  variables  that 

get  replaced  with  appropriate  values  by 
stage. OSAM. run. R  code 

dfOSAM.R 

Generates  the  appropriate  factors. csv  file  used  by 
dfosam.exe  using  the  factor.map.csv  file,  the  baseyear, 
startyear,  and  DP  number  as  input 

dfosam.exe 

short  VFP9  executable  that  updates  the  loss_adj.dbf  file 
based  on  the  factors. csv  file 

dfOSAM.bat 

calls  dfOSAM.R,  dfosam.exe  and  the  OSAM  executable  in 
proper  order  of  execution  on  the  compute  node 
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